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ENHANCING  FIRE  CONTROL  DECISION-MAKING  WITH  THE  PATRIOT  COGNITIVE 
SKILLS  TRAINER:  DEVELOPMENT  AND  VALIDATION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

In  this  research  project,  we  developed  and  validated  a  desktop  training  module  to  enhance 
fire  control  decision-making  and  problem  solving  skills  for  the  Phased- Array  Tracking  Radar  to 
Intercept  on  Target  (Patriot)  surface-to-air  missile  system  air  defense  launch  control 
crewmembers.  The  intent  was  to  provide  training  to  familiarize  less  experienced  air  defenders 
with  the  types  of  complex  problem  solving  and  reasoning  skills  that  expert  air  defenders  bring  to 
bear  in  making  their  engagement  decisions. 

Procedure: 

The  research  was  executed  in  three  phases.  In  the  first  phase,  we  identified  the  training 
needs  of  Patriot  air  defense  artillery  (ADA)  crewmembers  specific  to  tasks  relying  on  complex 
cognitive  skills,  i.e.,  decision-making  and  problem  solving  in  the  Patriot  operational  context.  In 
the  second  phase,  we  developed  training  to  address  these  identified  needs.  This  was 
accomplished  based  on  applying  the  ADDIE  (Analysis,  Design,  Development,  Implementation, 
and  Evaluation)  training  development  process,  with  in-depth  input  from  Patriot  subject  matter 
experts  (SMEs).  The  final  training  design  relied  on  identified  decision-triggering  events  taking 
place  within  a  hypothetical  air  defense  mission.  The  materials  developed  were  based  on 
unclassified  sources  and  focused  on  developing  Patriot  crewmembers  fire  control  decision¬ 
making  skills  over  training  specific  technical  operations  of  the  Patriot  system.  In  the  final  phase 
of  the  research,  the  Patriot  Cognitive  Skills  Trainer  (PCST)  was  validated  with  Soldiers  (Tactical 
Control  Assistants  (TCAs)  and  Warrant  Officers)  at  the  108th  ADA  BDE  at  Fort  Bragg,  NC. 

Findings: 

When  training  with  the  PCST,  Soldiers  were  engaged  and  found  the  training  beneficial, 
although  the  training  did  not  rely  on  simulating  the  sensitive  capabilities  and  operations  of  the 
Patriot  system.  High  fidelity  simulation  of  classified  systems  may  not  be  necessary  to  train  the 
cognitive  skills  required  in  conducting  operations  with  the  systems.  What  is  required  is  to 
provide  scenarios  that  viably  replicate  the  types  of  problems  that  would  arise  when  those  systems 
are  put  into  operation.  In  this  case,  we  developed  Patriot  mission  scenarios  and  problem  sets 
around  a  hypothetical  air  defense  of  friendly  forces  mission,  using  the  geography  of  a  location  in 
the  Southeastern  United  States.  Enemy  capabilities  and  threats  were  derived  from  open  source 
materials  and  were  intended  to  represent  what  air  defenders  could  face  in  any  number  of  real 
world  situations.  The  PCST  appeared  to  be  effective  in  training  Soldiers  on  decision-making  and 
problem  solving  skills  related  to  employment  of  the  Patriot  system  in  an  air  battle.  Soldiers 
reported  skill  improvement  in  six  out  of  1 1  areas  evaluated.  The  remaining  five  of  1 1  areas 
evaluated  showed  improvement,  but  were  not  statistically  significant. 


IV 


Utilization  and  Dissemination  of  Findings: 

The  results  of  the  research  were  briefed  to  the  U.S.  Army  Air  Defense  Artillery  School 
leadership  at  Fort  Sill,  OK,  in  Oct  2016.  Copies  of  the  training  product  have  been  provided  to 
Air  Defense  Artillery  School  (Fort  Sill,  OK),  instructors  at  the  ADA  Warrant  Officer  Advanced 
Course  (Fort  Sill,  OK),  simulation  operators  with  the  Standardized  Patriot  Evaluation  and 
Assessment  Reporting  (SPEAR)  team  (Fort  Bliss,  TX),  the  32nd  Army  Air  and  Missile  Defense 
Command  (AAMDC;  Fort  Bliss,  TX),  and  the  108th  Air  Defense  Artillery  Brigade  (Fort  Bragg, 
NC).  This  research  and  a  training  demo  were  presented  at  the  Department  of  Defense  Fab  Day 
Conference  at  the  Pentagon,  Washington,  DC,  in  May  2017. 
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Enhancing  Fire  Control  Decision-Making  with  the  Patriot  Cognitive  Skills 

Trainer:  Development  and  Validation 

Introduction 

The  Army  Air  Defense  Artillery  (ADA)  leadership  recognizes  that  new  ADA  Soldiers 
need  to  develop  complex  cognitive  skills  to  execute  air  defense  missions  effectively. 1  Even  so, 
most  air  defenders  are  trained  in  a  way  that  emphasizes  system-specific  technical  knowledge  and 
skills  over  cognitive  skills  related  to  decision-making  and  problem  solving  (Stallings,  Graves,  & 
Blankenbeckler,  in  press).  Often,  expert  air  defenders  have  honed  their  decision-making  and 
problem  solving  skills  over  the  course  of  a  career.  The  question  that  this  research  sought  to 
answer,  then,  is  how  this  learning  process  may  be  enhanced  for  ADA  Soldiers  who  have  not  yet 
gained  the  experience  of  an  expert.  In  this  research  project,  we  developed  and  validated  a 
desktop  training  module  to  enhance  fire  control  decision-making  and  problem  solving  skills  for 
air  defense  launch  control  crewmembers  on  the  Phased- Array  Tracking  Radar  to  Intercept  on 
Target  (Patriot)  surface-to-air  missile  system.  The  intent  was  to  provide  training  to  familiarize 
less  experienced  Patriot  air  defenders  with  the  types  of  complex  problem  solving  and  reasoning 
skills  that  expert  Patriot  air  defenders  bring  to  bear  in  making  their  engagement  decisions. 

The  Context  of  the  Research  Problem 

Patriot  air  defenders  often  face  complex  decision-making  tasks  under  conditions  of 
degraded  and/or  incomplete  information.  Moreover,  various  functions  of  the  Patriot  system  can 
be  automated,  with  the  operator  intervening  only  to  break  off  an  engagement.  For  more  than  a 
decade,  researchers  and  air  defense  leaders  have  been  raising  concerns  about  the  role,  scope,  and 
limitations  of  human  operators  within  highly  automated  systems  like  Patriot  (see  Hawley,  Mares, 
&  Giammanco,  2006).  Specific  issues  concern  supervisory  control,  problem  solving,  and 
decision  making  when  operating  complex  and  lethal  systems  in  ambiguous  environments. 
Hawley  and  Mares  (2007)  pointed  out  that  the  evolution  of  Patriot  crewmembers  from  traditional 
operators  to  supervisors  of  automated  processes  requires  changes  in  system  design,  decision  aids, 
training,  and  professional  development. 

Vigilance  and  situational  awareness  became  critical  issues  in  Patriot  research  following  a 
fratricide  incident  that  occurred  during  Operation  Iraqi  Freedom  (OIF)  I  (Hawley,  2006;  2008; 
2009).  Out  of  the  11  engagements  of  the  Patriot  system  during  OIF  I,  nine  (82%)  were 
successful  (Hawley,  2009).  The  two  unsuccessful  engagements  ended  in  fratricides.  One 
involved  a  U.S.  Navy  F/A-18  Hornet  that  was  misclassified  as  an  inbound  missile.  The  other,  a 
British  Royal  Air  Force  Tornado  was  misclassified  as  an  inbound  anti-radiation  missile  (ARM). 
Three  air  crewmembers  were  casualties  in  these  engagements. 

To  reduce  instances  of  fratricide,  Hawley  (2011)  has  emphasized  that  crews  must  have 
sufficient  understanding  of  system  operations  and  potential  limitations  of  the  data  they  are 
interpreting  in  order  to  decide  effectively  whether  to  launch  a  missile.  Crews  also  require  the 
expertise  to  understand  the  contextual  meaning  of  the  information  they  are  interpreting  to  make  a 


1  See  Appendix  A  for  a  discussion  of  the  contemporary  ADA  operational  context. 
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fire  decision.  Hawley  (201 1)  concluded  that  emphasizing  reliance  on  automated  technology  over 
Soldier  performance  significantly  contributed  to  the  problems  experienced  during  OIF.  He 
further  emphasized  that  current  training  practices  and  standards  would  not  meet  the  challenge  of 
developing  expertise  in  technology- intensive  systems.  Fratricides  are  the  most  demonstrative 
and  salient  indicator  of  the  training  problem  this  research  sought  to  address.  This  research  was 
intended  to  begin  to  address  these  issues  by  focusing  on  developing  and  validating  training  to 
support  enhanced  situational  awareness,  information  management,  and  decision  making  in 
complex  air  defense  engagements. 

The  Design  of  the  Research 

The  research  project  was  executed  in  three  phases.  The  end  goal  was  to  develop  and 
validate  a  desktop  training  module  to  enhance  fire  control  decision-making  and  problem  solving 
skills  in  Patriot  crewmembers.  In  the  first  phase,  we  sought  to  identify  deficiencies  in  existing 
Patriot  individual  and  crew  training  that  may  contribute  to  shortcomings  in  the  development  of 
complex  decision-making  and  problem  solving  skills.  In  the  second  phase,  we  identified  the 
skills  that  need  to  be  trained  and  selected  training  techniques  and  methods  to  incorporate  into  a 
prototype  training  tool:  the  Patriot  Cognitive  Skills  Trainer  (PCST).  Finally,  we  validated  the 
PCST  with  ADA  crewmembers. 

To  define  deficiencies  in  existing  Patriot  training,  we  reviewed  the  research  literature 
related  to  decision  making  and  conducted  interviews  with  ADA  Patriot  subject  matter  experts 
(SMEs).  SMEs  were  accessed  through  the  Air  Defense  School  at  Fort  Sill,  OK,  and  consisted  of 
trainers,  leaders,  and  training  developers  as  well  as  Soldiers,  leaders,  and  trainers  in  operational 
units.  This  phase  of  the  research  was  focused  on  defining  the  problem  and  examining  potential 
solutions.  This  included  (a)  identifying  current  institutional  and  unit  training  of  Patriot 
engagement  control  station  (ECS)  crewmembers  addressing  cognitive  skills  development,  and 
(b)  documenting  specific  situations  and  examples  of  complex  decision-making  situations  faced 
by  Patriot  ECS  crewmembers.  The  ECS  is  the  fire  control  unit  for  the  Patriot  system.  It  is 
usually  manned  by  three  Soldiers,  including  a  Tactical  Control  Officer  (TCO),  a  Tactical  Control 
Assistant  (TCA),  and  a  Communications  Operator.  Fire  control  decisions  typically  are  made  by 
the  TCO  with  support  from  the  TCA  and  Communications  Operator. 

The  research  team  examined  various  training  designs  and  methods  to  specify  the  features 
of  training  that  would  provide  the  context  and  feedback  essential  to  developing  crewmembers’ 
decision-making  skills.  The  training  scenarios  were  matched  with  instructional  design  concepts 
that  applied  principles  of  learning  and  the  Analysis,  Design,  Development,  Implementation,  and 
Evaluation  (ADDIE)  model  to  create  the  PCST  in  an  interactive  multimedia  instructional  format 
(Morrison,  Ross,  Kemp,  &  Kalman,  2010). 2  Key  to  the  design  was  an  effective  mechanism  for 
providing  feedback  to  the  learner.  Moreover,  the  training  needed  to  be  focused  not  on 
procedural  solutions,  but  instead  on  solutions  that  engaged  learners  in  information  management, 
prioritization,  interpretation,  and  reasoning.  Finally,  the  prototype  tool  would  need  to  provide 
trainers  and  learners  with  a  clear  path  to  developing  progressively  more  complex  decision¬ 
making  skills. 


2  ADDIE  is  an  instructional  design  framework  used  as  a  descriptive  guideline  for  building  effective  training  and 
performance  support  tools. 
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In  the  final  phase  of  the  research,  the  team  validated  the  tool  to  provide  recommendations 
for  ongoing  improvement  of  the  product  as  well  as  ways  to  refine  and  sustain  a  training  approach 
that  would  address  perishable  decision-making  skills.  The  PCST  was  tested  with  air  defenders  at 
the  108th  ADA  Brigade  and  Fort  Bragg,  NC.  Through  the  validation  test,  the  participating 
Soldiers  assisted  us  with  a  live  environment  assessment  of  the  PCST  and  an  examination  of 
perceived  skill  improvements. 


Phase  I:  Identifying  Training  Deficiencies 


Literature  Review 

The  research  team  conducted  a  two-pronged  review  to  gain  insight  into  the  problems 
associated  with  Patriot  fire  control  decision  making.  First  was  a  review  of  the  relevant  scientific 
literature,  including  documents  concerning  Patriot  and  related  air  defense  fire  control  decision 
making  in  combat  and  contingency  operations.  Significant  background  research  was  available 
due  to  the  Patriot  Vigilance  Project  (Hawley,  2009).  Their  efforts  focused  on  crewmember 
vigilance  and  situational  awareness  in  highly  automated  air  defense  battle  command  systems. 
The  materials  from  the  Patriot  Vigilance  Project  addressed  the  fratricide  incidents  from  OIF  I 
that  related  to  information  and  displays  available  to  the  crews  and  the  crews’  decision-making 
process. 

Training  Interviews  and  Observations 

The  second  prong  examined  how  Patriot  ECS  crewmembers  are  prepared  and  trained  for 
the  operational  environment.  This  review  included  both  institutional  and  in-unit  training. 

Air  Defense  School.  The  research  team  interviewed  25  trainers  and  leaders  at  the  Air 
Defense  School,  Fires  Center  of  Excellence,  Fort  Sill,  OK,  including: 

•  Senior  warrant  officer  leaders  at  the  ADA  School. 

•  ADA  School  Course  Managers  and  Course  Developers. 

•  ADA  School  leaders  and  instructors  representing: 

o  Air  Defense  Artillery  Basic  Officer  Leader  Course  -  Branch  (ADA  BOLC-B) 
o  Warrant  Officer  basic  and  advanced  courses  for  140E  -  Air  and  Missile  Defense 
(AMD)  Tactician/Technician 

o  Noncommissioned  Officer  (NCO)  leader  and  professional  development  courses  - 
the  Advanced  Leaders  Course  and  Senior  Leaders  Course  for  military 
occupational  specialties  (MOSs)  14E  Patriot  Fire  Control  Enhanced 
Operator/Maintainer  and  14T  Patriot  Launching  Station  Enhanced 
Operator/Maintainer 
o  Patriot  Master  Gunner  Course3 


3  The  Patriot  Master  Gunner  Course  is  an  intense  10-week  course  for  14T  and  14E  Staff  Sergeants  through  Master 
Sergeants.  It  provides  Patriot  NCOs  with  the  ability  to  develop,  implement,  and  evaluate  Patriot  gunnery  training 
strategies  that  will  take  units  through  Gunnery  Tables  I  thru  XII.  They  become  experts  on  all  aspects  of  the  Patriot 
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o  Patriot  Top  Gun  Course4 

Operational  Units.  Additionally,  the  research  team  interviewed  30  trainers  and  leaders 
from  operational  units,  batteries,  and  battalions  at  two  installations  as  well  as  their  controlling 
and  certifying  headquarters.  These  groups  included: 

•  Trainers  assigned  to  a  Patriot  battalion  who  had  recently  returned  from  an  operational 
deployment  (representatives  included  the  battalion  Standardization  NCO,  and  three 
battery  trainers  (one  officer  and  two  NCOs).  Of  note,  all  had  experience  from  the 
previous  deployment. 

•  The  staff  of  the  32nd  Army  Air  and  Missile  Defense  Command  (AAMDC)  Standardized 
Patriot  Evaluation  and  Assessment  Reporting  (SPEAR)  Team.5  This  unique  Team 
consisted  of  a  core  cadre  from  32nd  AAMDC,  a  group  of  highly  skilled  contractors  (Wyle 
Laboratories,  Inc.;  all  were  retired  Patriot  senior  warrant  officers  and  NCOs)  who  operate 
the  supporting  simulation  systems,  and  hand-picked  Patriot  SMEs  (senior  warrant 
officers  and  NCOs)  drawn  from  the  Patriot  battalions  of  the  32nd  AAMDC’ s  ADA 
Brigades. 

•  Trainers  and  training  staff  assigned  to  an  AMD  Brigade  with  Patriot  battalions  preparing 
for  two  staggered  overseas  deployment  missions.  This  group  included: 

o  The  Standardization  NCO  from  the  brigade  staff; 

o  The  Standardization  Officer,  fire  direction  officer  in  charge  (OIC),  and  four 
battery  trainers  from  one  Patriot  battalion;  and 
o  The  Standardization  NCO,  fire  direction  OIC,  and  three  battery  trainers  from  the 
second  Patriot  battalion. 

Finally,  the  research  team  also  examined  initial  entry  training  and  in-unit  preparations  and 
training. 

Initial  Entry  Patriot  Crewmember  Training.  Soldiers  are  first  prepared  for  Patriot 
duties  during  initial  entry  training.  The  14E  (Patriot  Fire  Control  Enhanced 
Operator/Maintainer)  advanced  individual  training  (AIT)  is  a  20  week  course,  introducing 
Soldiers  to  the  Patriot  system.  The  14E  AIT  is  focused  on: 

•  Operating  Patriot  technology  and  missile  systems, 

•  Placement  and  emplacement  of  the  Patriot  subsystems, 

•  Status  reporting, 

•  Maintenance  on  coordinate,  communications,  and  target-identification  systems, 


weapons  system  technical  and  tactical  employment  capabilities  and  limitations,  able  to  perform  and  analyze  Air  and 
Missile  Defense  Designs.  The  course  has  a  72%  graduation  rate  and  awards  the  additional  skill  identifier  T4. 

4  The  Patriot  Top  Gun  Course  is  an  intense  6-week  open  to  selected  promotable  First  Lieutenants,  Chief  Warrant 
Officers  2  (CW2s)  and  above.  Captains,  and  some  Majors.  The  Course  trains  the  Patriot  system  as  well  as 
integrated  missile  and  air  defense  systems  and  prepares  the  graduate  to  plan  and  design  missile  defenses  and  to 
advise  commanders  on  all  aspects  of  the  weapon  system,  missile,  radar,  as  well  as  threat  system  capabilities. 

5  The  32nd  AAMDC  SPEAR  Team  serves  as  a  rigorous,  third  party  mission  certification  element.  They  assure 
attainment  of  appropriate  levels  of  task  performance  prior  to  the  deployment  of  units  to  overseas  peacetime 
contingencies  and  certifies  units  outside  of  these  contingency  rotation  missions  for  their  wartime  missions. 
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•  Evaluating  intelligence  and  identifying  targets, 

•  Establishing  communications  with  tactical  systems,  and 

•  The  basics  of  air  defense  tactics  and  battle  strategy. 

In  later  phases  of  AIT,  Soldiers  are  introduced  to  the  screens,  control,  and  data  in  the  ECS  and 
Information  Coordination  Central  (ICC).  This  introduction  is  with  both  actual  hardware  and 
simulations.  On  completing  AIT,  most  14E  Soldiers  are  assigned  to  duties  in  a  Patriot  battery’s 
fire  control  platoon,  battery  command  post,  or  other  units  in  a  battalion.  A  smaller  number  are 
assigned  to  the  ECS,  ICC,  or  other  battle  management  duties. 

Warrant  Officer  Training.  AMD  Tactician  and  Technician  Warrant  Officers  (140E) 
are  vetted  extensively  prior  to  selection,  meeting  many  rigorous  prerequisites  before  they  are 
appointed  as  WOls.  The  140E  warrant  officer  candidates  are  selected  from  among  mid-career 
NCOs  (normally  SSGs  or  higher)  who  have  distinguished  themselves  with  respect  to  their 
knowledge  and  skill  with  the  Patriot  system.  A  potential  warrant  officer  also  has  at  least  two 
years  leadership  experience  in  a  feeder  MOS  such  as  14E,  14H,  14T,  or  94S.  Additional 
prerequisites  include  college  credits  in  English  and  Mathematics.  It  is  preferred  that  warrant 
officer  appointees  have  completed  the  Patriot  Master  Gunner  Course  and  have  served  at  least  two 
years  as  a  TCA,  Tactical  Director  Assistant  (TDA),  or  System  Maintenance  NCOIC.  Most 
applicants  meet  or  exceed  these  prerequisites. 

The  Warrant  Officer  Basic  Course  consists  of  35  weeks  of  classroom  and  practical 
training  on  tactics,  maintenance,  and  logistics  with  the  Patriot  system  and  subsystems.  The 
tactics  phase  of  the  course  is  six  to  seven  weeks  and  centers  on  Patriot  Gunnery  Tables  I-IV. 

The  WOls  must  score  at  least  80%  on  all  exams,  and  achieve  at  least  an  80%  on  the  Reticle  Aim 
Level  (RAL)  5  practical  exam  to  receive  Patriot  Basic  Gunnery  Certification  before  they 
graduate. 

Figure  1  depicts  mission  readiness  in  relation  to  the  Patriot  Gunnery  Table  Certifications. 
The  Reticle  Aim  Level  (RAL)  system  is  used  to  ensure  that  crewmembers  master  individual 
skills  that  support  collective  performance.  This  system  of  certifications  keeps  crewmembers 
from  attempting  procedures  or  air  battle  tasks  that  are  beyond  their  skill  level.  Gunnery  Tables 
guide  training  by  providing  checkpoints  for  required  individual  and  crew  skills.  In  certification 
drills,  crews  will  often  repeat  specific  drills  until  they  have  demonstrated  proficiency.  This 
approach  tends  to  develop  cognitive  skills  related  to  remembering,  understanding,  and  applying 
information  and  procedures  to  complete  successfully  the  gunnery/RAL  task  being  assessed. 
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Patriot  Gunnery  Tables/  Reticle  Aim  Levels  (RALs): 
Training  Proficiency  and  Mission  Readiness 
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Figure  1.  Mission  Readiness  in  Relation  to  Patriot  Gunnery  Tables.  Adapted  from  Patriot 
Gunnery  Tables/Reticle  Aim  Levels  (RALs):  Training  Proficiency  and  Mission  Readiness. 


Deployment/ 

MissioUPeady 


Level 

Table 

Subject  Matter 

Corresponding 

RAL 

Basic  Gunnery 
Tables 

1 

Basic  System  Skills 

II 

Ready-for-Action  Drills 

III 

Basic  Air  Battle  Management 

1.2.  3.4.5 

IV 

Basic  Gunnery  Certification 

5  * 

Intermediate 
Gunnery  Tables 

V 

Air  Battle  Management/Missile  Reload 

6.  7,  8.  9 

VI 

Daytime  March  Order  and  Emplacement 

VII 

Precertification  Tables  V  and  VI 

1  through  10. 11 

VIII 

Intermediate-Level  Gunnery  Certification 

11 

Advanced 
Gunnery  Tables 

IX 

Advanced-Level  Air  Battle  Management  and  Missile 
Reload 

12. 13 

X 

Nighttime  MO&E  Under  Varying  CBRN  Conditions 

XI 

Precertification  Tables  IX  and  X 

14. 16,  17 

XII 

Advanced-Level  Gunnery  Certification 

17 

Table  2-1  from  FM  3-01.86  Air  Defense  Artillery:  Patriot  Brigade  Gunnery  Program, 

dated  23  Dec  08w/CHl. 

RAL  5  Certification  equates  to  a  PATRIOT  crew  Learner's  Permit. 


Commissioned  Officer  Training.  New  Lieutenants  come  to  ADA  from  various 
commissioning  sources.  These  sources  include  the  Reserve  Officers  Training  Corps,  the  U.S. 
Military  Academy,  and  Officers  Candidate  School.  Commissioning  requires  that  all  have  a 
bachelor’s  or  higher  degree.  Few  have  AMD  experience.  The  ADA  Basic  Officer  Leader 
Course  -  Branch  (BOLC-B)  is  two  phased.  Phase  1  (6  weeks  and  3  days)  focuses  on  common 
core  program  lessons,  leadership  and  planning,  and  common  warfighting  skills.  Phase  2(12 
weeks)  provides  branch  and  career  management  field  technical  content  for  warfighting  skills  and 
ADA,  focusing  on  AMD  weapon  systems.  This  includes  Short  Range  Air  Defense  Platoon 
Leader,  Patriot  Platoon  Leader,  Patriot  TCO,  and  AMD  Operational  Exercises.  The  Patriot 
Platoon  Leader  module  introduces  new  officers  to  the: 


•  Patriot  gunnery  program, 

•  Reconnaissance,  selection,  and  occupation  of  position  for  the  Patriot  system,  subsystems, 
and  support  elements, 

•  March  order  and  emplacement  for  the  Patriot  system,  subsystems,  and  support  elements, 
and 

•  Responsibilities  of  the  TCO. 

The  familiarization  training  for  Patriot  TCO  includes: 
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•  Identifying  and  defining  the  function  of  ECS  switches,  controls,  indicators,  and  tabs, 

•  ECS  initialization, 

•  Radar  mapping, 

•  Patriot  reports, 

•  Fix  or  fight  fault  assessments  and  maintenance, 

•  Ready  for  action  drills, 

•  System  reorientation,  and 

•  Engagement  decisions  and  weapon  assignment  processing. 

BOLC-B  Patriot  training  tends  to  be  high  level,  since  the  majority  of  new  Lieutenants  (LTs)  are 
assigned  outside  Patriot  units.  The  training  is  intended  to  orient  and  familiarize  LTs  with  Patriot 
associated  duties.  The  officer’s  course  is  the  shortest  of  the  three  basic  courses,  with  only  a 
fraction  of  training  focused  on  Patriot. 

In-Unit  Patriot  Crewmember  Training.  Duties  at  a  Patriot  unit  of  assignment  vary. 
Duty  assignments  are  based  on  unit  needs  and  vacancies,  command  priorities,  individual 
aptitude,  career  requirements,  individual  experience,  and  Soldiers’  preferences.  The  structure 
and  intensity  of  Patriot  Fire  Unit  (FU)  training  goes  beyond  that  of  most  Army  units.6  Training 
certifications  are  closely  tied  to  evaluations  of  unit  readiness.  To  maintain  readiness,  a  Patriot 
FU  must  have  at  least  two  certified  ECS  crews.  Both  the  primary  and  secondary  crews  should  be 
collectively  certified  for  Gunnery  Table  VIII  (the  gate  for  intermediate  gunnery)  and  Table  XII. 
Sustained  operations,  with  8-hour  shifts,  could  require  three  crews.  For  peacetime  operations, 
the  third  crew  is  considered  a  sustainment  crew,  which  would  also  meet  the  minimum 
certification  requirements. 

Each  ECS  crew  is  made  up  of  three  personnel.  The  TCO  is  typically  a  Lieutenant  or 
Warrant  Officer.  Several  warrant  officers  we  interviewed  indicated  that  they  had  served  as  a 
TCO.  In  our  observations,  however,  only  Lieutenants  were  in  the  TCO  position.  Interviews  also 
indicated  that  senior  NCOs  sometimes  serve  as  TCOs.  NCOs,  Sergeants  (E-5s)  or  Staff 
Sergeants  (E-6s),  are  normally  in  TCA  positions.  During  our  observations,  Specialists  (E-4s) 
were  serving  as  TCAs.  In  one  situation,  a  Private  (E-2)  routinely  manned  a  TCA  position.  We 
observed  Specialists  (E-4s)  and  Privates  First  Class  (E-3s)  in  the  Communications  Operator 
position;  no  NCOs  were  observed  in  this  position.  Table  1  describes  this  typical  crew 
composition. 


6  Army  Field  Manual  3-01.86,  Air  Defense  Artillery:  Patriot  Brigade  Gunnery  Program  is  the  guide  of  training 
standards  and  requirements  for  Patriot  units  at  all  echelons  (Department  of  the  Army,  2008b). 
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Table  1 

Patriot  Engagement  Control  Station  (ECS)  Crew  Composition 

Duty  Position 

Crewmember 

Tactical  Control  Officer 
(TCO) 

ADA  Lieutenant  (14A)  or  AMD  Patriot  Systems  Technician, 
Warrant  Officer  (140E) 

Tactical  Control  Assistant 
(TCA) 

Patriot  Fire  Control  Enhanced  Operator  NCO 
(14E20  -  14E30) 

Communications  Operator 

Multichannel  Transmission  System  Operator  (25Q10  -25Q20) 
or  Network  Switching  System  Operator  (25F10  -  25F20) 

Note:  Units  develop  and  maintain  three  crews  to  assure  effective  24-hour  operations. 


The  TCO  is  responsible  for  crew  and  ECS  operations.  The  TCO’s  duties  are  to  protect 
friendly  aircraft  in  their  sector,  monitor  the  network,  maintain  voice  communication  with  ICC 
and  other  batteries,  as  well  as  with  higher  echelon  networks.  The  TCO  updates  the  system, 
assuring  accurate  input,  activation,  and  deactivation  of  AMD  messages,  orders,  and  plans.  These 
include: 

•  Area  air  defense  plans  and  changes, 

•  Special  instructions  for  theater  specific  air  defense  operations, 

•  Air  tasking  orders  as  specified  within  the  air  defense  plan, 

•  Airspace  control  orders  coordinated  by  higher  headquarters,  and 

•  Tactical  operations  data. 

The  TCA  assists  the  TCO  by  assuming,  as  needed,  TCO  duties.  We  observed  TCAs 
engage  tracks  as  instructed  or,  after  confirming  tracks  as  hostile,  handing  them  off  to  the  TCO. 
The  TCA  brings  the  launchers  from  “standby”  to  “operate”  and  presses  the  “engage”  switch  to 
initiate  a  missile  launch.  The  TCA  also  monitors  launcher  and  missile  status,  and  missile 
expenditures  and  impacts. 

The  third  member  of  the  crew,  the  Communications  Operator,  is  technically  not  an  air 
defender.  The  Communications  Operator  is  responsible  for  receiving,  processing,  and  submitting 
reports,  and  maintaining  a  log  of  critical  actions.  The  TCO,  TCA,  and  Communications  Operator 
train  to  function  as  team.  In  proficient  crews,  the  TCO  and  TCA  are  cross-trained  on  each 
other’s  duties.  The  research  team  frequently  observed  TCOs  and  TCAs  working  over  each 
other’s  shoulders  when  required.  Communications  Operators  were  observed  to  assist  in  the 
synergy  between  TCO  and  TCA.  On  occasion,  the  Communications  Operator  relieved  some 
workload  from  TCO  or  TCA,  assisting  with  or  assuming,  a  lower  priority  task.  These  tasks 
included  alerting  the  crew  to  status  changes,  critical  situations,  or  reminding  them  of  required 
actions. 
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The  Patriot  unit  is  primarily  responsible  for  training  Basic  Gunnery  and  RAL  1  through  4 
(refer  to  Figure  1),  although  there  is  no  official  duty  position  for  Battery  Trainers.  Our 
interviews  indicated  that  all  batteries  had  appointed  battery  trainers,  who  served  in  this  position 
as  an  additional  duty.  Battery  trainers  would  manage,  coordinate,  administer,  and  conduct 
training  for  new  arrivals  and  crewmembers  requiring  certification.  We  observed  that  the 
appointment  of  a  battery  trainer  was  based  on  the  needs  of  and  vacancies  within  the  battery, 
demonstrated  aptitude  of  the  appointee,  command  priorities,  individual  experience,  and 
sometimes  the  preferences  of  the  appointee.  In  some  cases,  commanders  had  taken  particular 
care  to  ensure  a  high  quality,  very  knowledgeable  lieutenant  or  NCO  assumed  this  additional 
duty.  We  did  not  observe  any  warrant  officers  in  these  positions.  Most  often,  battery  trainers 
were  senior  First  Lieutenants  or  NCOs,  who  were  Sergeants  (E-5)  through  Sergeants  First  Class 
(E-7).  In  one  case,  our  research  uncovered  a  Specialist  (E-4)  who  had  been  appointed  a  battery 
trainer  due  to  unit  turbulence  and  his  “knack  for  teaching.”7  Many  battery  trainers  received  little 
if  any  additional  training  to  prepare  for  the  position.  However,  we  did  interview  some  NCOs 
serving  as  battery  trainers  who  were  graduates  of  the  Patriot  Master  Gunner  course.  If  they 
received  additional  training,  a  battalion  Standardization  Officer  or  NCO  often  provided  it.  To 
conduct  training,  battery  trainers  often  relied  on  their  own  experiences,  materials  existing  in  the 
battery  or  battalion,  or  materials  they  developed  on  their  own. 

SPEAR  Certification  Training.  The  research  team  observed  two  SPEAR  training  and 
certification  events  for  Patriot  battalions  preparing  for  overseas  missions.  The  two  missions 
varied  in  areas  of  operation,  geography  and  air  space,  potential  threat,  and  controlling  integrated 
air  and  missile  defense  (IAMD)  command  structures  and  procedures.  As  a  result,  each  unit 
observed  had  divergent  rules  of  engagement  (ROE)  and  tactics,  techniques,  and  procedures 
(TTP),  adjustments  to  existing  tactical  standard  operating  procedures,  and  routine  operations 
procedure.  The  unique  characteristics  of  each  unit’s  mission  were  incorporated  into  the 
framework  and  standards  of  the  SPEAR  assessment  and  training. 

Each  SPEAR  event  included  three  to  four  time-collapsed  simulated  air  battles  per  day, 
with  each  battle  lasting  between  20  and  45  minutes.  Air  battles  were  followed  by  an  after-action 
review  (AAR).  Communications  links  were  realistically  simulated,  involving  role  players  acting 
as  higher  headquarters  and  IAMD  controlling  agencies.  Impartial  observers  tracked  and  assessed 
how  each  crew  performed.  Performance  observations  were  made  in  the  ECS  and  ICC  vans  as 
well  as  on  simulation  terminals,  listening  to  radio  communications,  and  reviewing  recorded 
communications  in  text  and  on  voice  circuits.  AARs  were  conducted  both  on-site,  sometimes 
with  an  observer  facilitating  the  crew  discussions,  and  as  a  group,  with  all  training  participants 
attending.  The  AARs  covered  performance  standards,  challenges,  areas  for  improvement  as  well 
as  individual  and  crew  strengths.  Identified  challenges  were  addressed  in  subsequent  air  battles. 
Scenarios  were  modified  to  provide  remediation,  practice,  and  reinforcement.  SPEAR 
certification  events  may  last  up  to  two  weeks  and  include  multiple  visits.  For  example,  some 
certifications  begin  with  an  intense  training  session,  followed  by  a  break  for  in-unit  training, 
remediation,  and  practice,  and  then  are  concluded  with  the  SPEAR  certification  session. 


7  The  “Specialist’s  knowledge  sought  as  battery  trainer”  article  (2013)  describes  a  Specialist  (E-4)  with  the  10lh 
AAMDC  serving  as  a  battery  trainer,  who  had  been  handpicked  for  the  position  because  of  his  highly  developed 
Patriot  knowledge  and  skills.  Accessed:  https://www.dvidshub.net/news/printable/113278 
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SPEAR  air  battle  scenarios  were  based  on  the  Patriot  Gunnery  Tables.  However,  SME 
evaluators  and  simulation  operators  incorporated  novel  but  plausible  situations  into  the  simulated 
air  battles.  The  intent  was  to  challenge  the  crews.  The  simulation  operators  were  very  adept  in 
orchestrating  the  novel  scenario  details.  They  were  able  to  introduce  on  short  notice  novel 
situations  as  teaching  points  when  requested  to  by  on-site  observers/evaluators.  Simulation 
operators  also  introduced  variations  to  scenarios  on  the  fly,  responding  to  comments  made  by 
participants  over  voice  communications  nets. 

The  SPEAR  training  we  observed  consisted  of  battalion  ICC,  the  ECS  crews  of  three  or 
more  FUs  (depending  on  crews  to  train),  and  in  one  case  a  battery  command  post.  Air  battle 
scenarios  included: 

•  Real  world  (potential)  enemy  and  coalition/friendly  air  and  missile  order  of  battle, 

•  Mission  area  of  operations  (both  terrain  and  airspace), 

•  Mission  area  unique  ROE,  restrictions,  and  limitations, 

•  TTP  modified  for  the  mission  area,  and 

•  Operations  under  the  unique  IAMD  command  and  control  (Joint  or  Combined) 
procedures  for  the  mission  area. 

Many  variations  were  introduced  into  the  simulations,  specific  to  the  units’  upcoming 
mission  areas.  These  included  theater  ballistic  missiles  (TBMs)  and  Air  Breathing  Threats 
(ABTs)  portraying  probable  threat  platforms,  and  tactics  and  techniques  that  the  enemy  could  be 
expected  to  employ.  Some  scenarios  incorporated  degraded  system  capabilities  and 
communications.  Other  scenarios  were  conducted  under  chemical,  biological,  radiological, 
nuclear,  and  explosives  (CBRNE)  conditions  while  wearing  protective  ensembles.  SPEAR  trains 
for  a  worst-case  eventuality;  should  adversaries  initiate  hostilities,  the  units  trained  in  SPEAR  are 
prepared.  The  air  battles  we  observed  allowed  crews  to  practice  high-intensity  AMD  operations, 
preparing  them  to  handle  enemy  threats  while  deployed. 

Developing  an  Unclassified  Set  of  Training  Scenarios 

The  research  team  considered  existing  training  scenarios  to  help  us  plan  how  we  would 
develop  the  PCST  to  elicit  and  develop  decision-making  skills.  We  examined  scenarios  used  in 
SPEAR  events  and  on  the  Patriot  Reconfigurable  Table  Top  Trainer  (RT3),  a  computer-based  air 
battle  simulator.  Due  to  classification  restrictions,  the  existing  training  scenarios  were  used  only 
as  a  model.  We  intended  the  PCST  to  be  unclassified.  Therefore,  real  world  missions  and 
locations  as  well  as  actual  air  and  missile  order  of  battle  and  system  capability  information  could 
not  be  included  in  the  product  we  were  planning  to  develop.  Yet,  the  training  product  still 
needed  to  be  realistic  and  engaging  for  our  intended  audience.  To  assist  in  our  effort,  the 
SPEAR  Team  staff  provided  copies  of  two  archived  unclassified  sources: 

•  An  unclassified  exercise  model  used  in  previous  joint  and  combined  training,  and 

•  A  status  monitor  handbook  developed  by  32nd  AAMDC  for  the  Patriot  software  build, 
Post-Deployment  Build-7,  providing  information  on  fix  or  fight  system  maintenance 
displays  and  faults. 
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The  SPEAR  Team  also  provided  recommendations  and  guidance  in  developing  realistic 
training  scenarios  while  avoiding  classification  issues  and  information  restrictions.  These 
included: 

•  Develop  a  general  situation  with  fictitious  combatants,  one  being  supported  by  the 
contingency  deployment  of  a  United  States  mythical  Joint  Task  Force. 

•  Frame  the  general  situation  in  a  developing  theater  with  limited  resources  and  tailored 
constraints.  Patriot  should  be  the  primary  missile  and  high  altitude  antiaircraft  protection 
for  U.S.  forces  as  well  as  the  host  nation  and  cooperating  coalition  forces. 

•  Select  a  location  and  area  of  operations  within  the  continental  United  States  away  from 
external  borders.  The  situation  should  have  no  implications  of  or  connections  to  a  real 
world  contingency  or  possible  homeland  defense  mission.  The  Washington,  D.C.  area 
for  example  should  be  avoided.  Keep  the  air  battle  over  land. 

•  Design  a  plausible  and  moderately  capable  enemy  air  and  missile  threat.  Build  a  simple 
threat  order  of  battle  composed  of  Cold  War  era  systems  and  weapons  that  have  seen  a 
wide  proliferation  through  foreign  military  sales.  This  order  of  battle  should  include 
aircraft,  TBMs,  cruise  missiles,  and  unmanned  aerial  vehicles. 

•  Draw  threat  system  capabilities  from  open  source,  unclassified  websites  and  technical 
sources  that  are  credible. 

•  Avoid  restrictive  ROE;  keep  the  air  battle  and  the  AMD  operations  of  the  force  in  a 
doctrinal  context. 


Results  of  the  Phase  I  Analysis  of  Training 

In  Phase  I  of  this  research,  the  research  team  analyzed  Patriot  ECS  crewmember  training 
in  the  context  of  scientific  literature  concerning  decision-making  and  critical  thinking  skills 
development.  This  process  was  intended  to  identify  shortcomings  in  current  training  for 
developing  complex  cognitive  skills.  We  sought  to  specify  what  scientific  findings  and  concepts 
held  promise  for  a  training  solution  based  on  the  current  training  and  working  environment. 

Identifying  Thinking  Skills  Required  for  Decision-Making  in  Patriot  Crews 

A  well-known  approach  to  classifying  cognitive  skills  is  Bloom’s  taxonomy  (Bloom, 
Englehart,  Furst,  Hill,  &  Krathwohl,  1956).  Bloom’s  taxonomy  differentiates  and  hierarchically 
arranges  thinking  skills  in  terms  of  higher-order  and  lower-order  skills.  Lower-order  thinking 
skills  relate  to  knowing,  comprehending,  and  applying  information;  higher-order  thinking  skills 
relate  to  analyzing,  synthesizing,  and  evaluating  information.  Each  higher-order  skill 
incorporates  those  of  a  lower-order.  For  example,  one  cannot  apply  knowledge  without  first 
knowing  it  and  comprehending  it. 

This  taxonomy  has  continued  to  develop  in  contemporary  psychological  research 
(Anderson  &  Krathwohl,  2001).  This  contemporary  take  on  Bloom’s  taxonomy  retains  the 
lower-order  to  higher-order  hierarchy  of  cognitive  skills,  but  transforms  Bloom’s  static  nouns 
into  verbs.  It  also  puts  ‘creating’  at  the  top  of  the  hierarchy,  rather  than  ‘evaluation.’  Figure  2 
illustrates  some  of  the  distinctions  between  the  early  and  contemporary  theories. 
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Higher-Order  Thinking  Skills 
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— ►  Knowledge 

Bloom  et  al.  (1956) 

Anderson  &  Krathwohl  (2001) 

Nouns 

Verbs 

Lower-Order  Thinking  Skills 


Figure  2.  Thinking  skills  hierarchy.  Adapted  from  Stallings,  Graves,  &  Blankenbeckler  (in 
press). 


Anderson  and  Krathwohl  (2001)  have  further  specified  cognitive  processes  into  six  levels 
and  19  categories.  Table  2  provides  a  synopsis  of  their  terminology.  The  updated  terminology 
can  be  more  easily  related  to  processes  in  military  operations,  including  those  related  to  crew 
operations  in  the  ECS  and  AMD  actions  and  engagements. 
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Table  2 

Contemporary  Terminology  Describing  Thinking  Skills 


Level 

Descriptive  Terms 

Additional  Examples 

Create 

Generating  -  hypothesizing 

Planning  -  designing 

Producing  -  constructing 

create,  invent,  compose,  predict, 
plan,  imagine,  propose,  devise, 
formulate,  combine,  originate, 
forecast,  invent,  produce,  assemble 

Evaluate 

Checking  -  coordinating,  detecting, 
testing 

Critiquing  -  judging 

select,  choose,  decide,  justify,  debate, 
verify,  argue,  recommend,  assess, 
discuss,  rate,  prioritize,  determine, 
critique,  criticize,  weigh,  value, 
estimate,  defend,  convince,  support, 
score 

Analyze 

Differentiating  -  discriminating, 
distinguishing,  focusing,  selecting 
Organizing  -  finding  coherence, 
integrating,  outlining,  parsing, 
structuring 

Attributing  -  deconstructing 

examine,  investigate,  separate,  take 
apart,  differentiate,  subdivide, 
deduce,  compare,  contrast,  infer, 
calculate,  monitor 

Apply 

Executing  -  carrying  out 

Implementing  -  using 

solve,  construct,  compute,  complete, 
make,  put  together,  change,  produce, 
calculate,  manipulate,  modify, 
demonstrate 

Understand 

Interpreting  -  clarifying,  paraphrasing, 
representing,  translating 

Exemplifying  -  illustrating, 
instantiating 

Classifying  -  categorizing,  subsuming 
Summarizing  -  abstracting, 
generalizing 

Inferring  -  concluding,  extrapolating, 
interpolating,  predicting 

Comparing  -  contrasting,  mapping, 
matching 

Explaining  -  constructing  models 

outline,  discuss,  distinguish,  predict, 
restate,  describe,  relate,  summarize, 
convert,  visualize,  describe,  sketch 

Remember 

Recognizing  -  identifying 

Recalling  -  retrieving 

tell,  list,  locate,  write,  find,  state, 
name,  identify,  label,  define, 
reproduce,  memorize,  select,  recite 

SME  comments  indicated  that  they  believed  new  ECS  crews’  decision-making  skills 
improve  with  time  and  experience.  Likely  due  to  the  complexity  of  the  Patriot  system,  most 
training  targets  improved  procedural  techniques  rather  than  decision-making  skills.  Procedures 
rely  on  lower-order  thinking  skills:  remembering,  understanding,  and  applying.  True 
decisions — requiring  crews  to  analyze  and  evaluate  information,  and  create  a  solution — were 
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infrequent.  Our  research  team  observed  that  situations  requiring  decisions,  other  than 
prioritizing  and  reprioritizing  actions,  occurred  once  or  twice  a  day.  That  is,  at  a  rate  of 
approximately  every  other  air  battle  scenario.  On  this  basis,  we  determined  that  a  training 
solution  that  compressed  time  and  provided  multiple  decision  points  could  provide  increased 
decision-making  opportunities  and  practice  of  higher-order  cognitive  skills.  This  compressed 
period  could  build  decision-making  experience,  if  combined  with  feedback  and  a  means  of 
showing  and  tracking  skill  improvement. 

A  long-standing  question  in  training  research  is  how  to  develop  learners’  conceptual 
frameworks  to  enable  them  to  use  what  they  have  learned  in  contexts  that  differ  from  training 
(Blume,  Ford,  Baldwin,  &  Huang,  2009).  When  learners  are  able  to  develop  a  more  abstract 
representation  of  knowledge,  they  are  more  likely  to  be  able  to  apply  it  in  real  world  contexts 
beyond  the  training  environment  (Bransford  &  Schwartz,  2001).  Mental  models  research 
concerns  both  how  an  individual  comes  to  represent  information  internally  as  well  as  how  a 
group  comes  to  share  an  understanding.  Research  on  teams  has  shown  that  when  they  share  a 
conceptual  framework,  they  tend  to  be  more  effective  as  they  are  more  able  to  anticipate  each 
other’s  needs  (Van  den  Bossche,  Gijselaers,  &  Segers,  2011).  Our  observations  indicated  that 
the  more  effective  ECS  crews  could  anticipate  each  other’s  actions  and  emerging  information 
needs  during  the  simulated  air  battles,  responding  to  the  present  context  of  the  air  battle. 

A  number  of  techniques  have  been  developed  to  enable  individuals  and  teams  to 
represent  and  communicate  about  their  conceptual  understandings  (e.g.,  mental  models). 
Individuals  and  teams  can  use  these  techniques  to  reflect  more  ably  on  their  own  and  their  team 
members’  thinking  processes.  These  techniques  include  (de  Bono,  1985;  Teach  for  America, 
2011): 

•  Visualizing  a  problem  by  diagramming  it, 

•  Separating  relevant  from  irrelevant  information, 

•  Seeking  reasons  and  causes, 

•  Justifying  solutions  to  problems, 

•  Seeing  more  than  one  side  of  a  problem, 

•  Weighing  sources  of  information  based  on  their  credibility, 

•  Revealing  assumptions  in  reasoning,  and 

•  Identifying  bias  or  logical  inconsistencies. 

During  our  observations  of  training,  we  noted  that  many  times,  crewmembers  would 
externalize  mental  processes  by  drawing  diagrams,  making  notes,  etc.,  using  a  wax  pencil  on  the 
scope.  A  problem  arises  because  crew  stations  in  the  ECS  van  have  little  extra  space  (see  Figure 
3).  These  extemalization  techniques  were  based  on  individual  preference  and  were  not  standard. 
The  workstation  design  afford  little  room  for  supplemental  checklists  or  information  displays. 
Visualization  of  the  problem,  diagraming  of  information,  or  comparing  information  would 
ultimately  need  to  rely  on  a  conceptual  framework.  In  the  design  of  our  decision-making  skills 
training,  we  determined  that  we  would  need  to  use  a  design  that  permitted  learners  to  compare 
information  and  to  determine  whether  pertinent  data  was  present  for  their  decision-making 
process.  In  this  way,  we  would  be  able  to  sequence  training  to  provide  scaffolding  for  mental 
processes  early  in  the  training,  then  shift  users  toward  greater  reliance  on  their  conceptual 
frameworks  later  in  the  training,  as  they  advanced. 
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ECS  Crewmember  Duty  Stations 


Figure  3.  ECS  Crewmember  Duty  Stations  (adapted  from  Balmefrezol,  2009) 

Types  of  Decision-Making  for  Patriot  Crews 

In  a  Patriot  context,  decisions  often  are  tied  to  actions,  such  as  engagement,  attack, 
resupply,  or  fault  identification  and  repair.  The  action  in  question  may  require  immediate 
execution,  or  it  may  address  an  anticipated  need  to  be  incorporated  into  a  plan  for  future  action. 

In  the  research  literature,  decision-making  is  often  described  from  two  perspectives:  normative 
and  descriptive.  Normative  decision-making  concerns  how  people  ought  to  make  decisions.  It 
assumes  that  our  decision-making  process  is  optimally  rational  and  mathematical,  but  that 
various  psychological  mistakes  bias  the  process.  The  idea  is  to  figure  out  the  mistakes  and 
inoculate  decision-makers  against  them.  Descriptive  decision-making  concerns  how  people 
actually  make  decisions  in  the  real  world.  It  seeks  to  document  the  decision-making  process 
from  an  ecological  perspective — how  do  people  actually  decide  in  normal,  everyday  settings. 
Here,  we  sought  a  middle-ground  approach  to  understand  how  decision-makers  in  the  ECS  van 
actually  make  decisions  as  well  as  how  their  process  could  potentially  be  improved. 

Under  ideal  circumstances,  TCOs  and  TCAs  will  develop  skills  with  both  intuitive, 
everyday  decision-making  and  more  systematic,  rational  approaches.  Different  situations  call  for 
different  types  of  decision-making  processes.  For  many  engagement  decisions,  the  speed  at 
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which  an  aircraft  or  missile  is  approaching  calls  for  a  rapid  decision-making  process.  Only  so 
much  information  can  be  gathered  and  weighed  before  a  decision  must  be  made.  There  is 
frequently  an  element  of  uncertainty  to  an  engagement  decision;  accuracy  and  completeness 
cannot  be  sacrificed  without  also  incurring  risk.  Failing  to  consider  all  the  available  pertinent 
factors  can  result  in  a  flawed  decision.  The  dichotomy  between  taking  the  time  to  be  methodical 
and  risking  protected  assets,  or  making  a  hasty  decision  and  risking  friendly  causalities, 
fratricide,  or  mission  failures  highlights  the  need  for  some  structure  in  all  ECS  decisions.  In 
developing  the  PCST,  we  sought  to  design  around  a  process  that  allowed  for  identification  of 
essential  information  to  mitigate  the  risk  of  overlooked,  ignored,  or  omitted  information  so  that 
the  result  of  one  decision  does  not  adversely  affect  the  conditions  or  outcome  of  another. 

Fratricide  is  a  frequently  cited  example  of  the  result  of  a  poor  engagement  decision. 
Losing  a  friendly  aircrew  is  a  terrible  tragedy.  A  fratricide,  however,  pales  in  comparison  the 
impact  a  hostile  TBM,  aircraft  bomb,  or  cruise  missile  delivering  a  weapon  of  mass  destruction 
can  have  on  a  friendly  unit,  troop  concentration,  or  urban  complex.  Recent  U.S.  combat 
experience  holds  no  example  of  this  type  of  decision;  one  that  let  a  hostile  system  get  through. 

Conclusions  from  the  Phase  I  Analysis 

Current  Patriot  training,  while  often  intense,  is  focused  on  procedural  skills.  It  exercises 
cognitive  skills  related  to  remembering,  understanding,  and  applying  information.  In  the 
cognitive  skills  hierarchy,  these  tend  to  be  lower-order  thinking  skills.  While  these  skills  support 
decision-making,  they  are  still  removed  from  what  ECS  crews  need  to  learn  in  order  to  be  most 
effective  at  the  decision-making  process. 

It  is  important  that  junior  enlisted  Soldiers,  new  warrant  officers,  and  newly 
commissioned  officers  establish  a  baseline  of  knowledge  and  familiarity  with  the  Patriot  system 
and  mission.  In  courses  such  as  AIT  or  BOLC-B,  only  the  most  basic  Patriot  procedures  and 
skills  are  introduced.  While  talent  and  achievement  are  recognized  and  encouraged  in  all 
training  programs,  mastery  of  the  Patriot  system  is  not  a  training  goal.  This  early-career  training 
assures  that  course  graduates  attain  the  minimum  standard  required  to  begin  more  intensive 
training  and  performance  in  a  Patriot  unit. 

The  institutional  training  for  officers,  warrant  officers,  or  enlisted  Soldiers  appears  not  to 
focus  specifically  on  broadening  critical  thinking  and  decision-making  skills  or  to  refine  their 
skills  as  trainers.  Generally,  these  Soldiers  and  junior  officers  are  trained  to  execute  procedures. 
While  lieutenants  receive  some  training  in  troop  leading  procedures  (TLP),  mission  analysis,  and 
developing  an  operations  order,  this  training  focuses  on  identifying  mission  tasks,  planning  to 
execute  them,  then  executing  them  in  the  context  of  a  specific  mission  and  the  stated 
commander’s  intent.  Seldom  do  missions  require  extensive  analysis.  Innovation  is  often 
discouraged  in  mission  accomplishment  as  “innovation”  by  an  inexperienced  junior  officer  can 
precipitate  disaster.  NCOs  train  Soldiers,  uphold  standards,  and  carry  out  or  execute  orders. 
Their  duties  seldom  require  more  than  basic  comparative  analysis  or  planning. 
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Critical  thinking  was  only  occasionally  observed  in  SPEAR,  gunnery  certification 
training,  and  in-unit  training.  Most  training  we  observed  was  procedural.  In  training, 
crewmembers’  actions  tended  to  focus  on: 

•  Proper  preparation  of  and  initialization  of  the  Patriot  system, 

•  Operation  of  the  Patriot  system, 

•  Use  and  proper  interpretation  of  switches,  controls,  indicators,  settings,  and  tabs  of  the 
ECS, 

•  Updating  the  file  holdings  and  databases  as  new  orders  and  special  instructions  changed 
the  air  battle  details  and  geometry,  and 

•  The  execution  standard  (or  mission  modified)  TTP. 

Some  missions  and  deployments  introduced  additional  ROE.  These  restrictive  rules 
frequently  required  modifications  of  existing  standing  operating  procedures  (SOPs)  and  TTP, 
dictating  very  exacting  permissions  and  procedures  for  specified  situations  and  engagements. 
These  became  rigid,  structured  procedures.  In  short,  restrictive  procedures  further  reduced  the 
latitude  and  decision-making  role  of  crewmembers,  reducing  some  potential  decisions  to 
procedural  tasks  or  gaining  approval  for  a  task  with  actions  dictated  by  direction  of  higher 
headquarters. 

Aside  from  threat  or  work  prioritization,  crew  activities  seldom  used  higher-order 
thinking  skills — such  as  analyzing,  evaluating,  or  creating — during  the  mock  air  battles.  In 
discussions  of  air  battle  scenarios,  Patriot  SMEs  rarely  agreed  on  all  aspects  of  critical  thinking. 
There  appeared  to  be  no  standard  set  of  best  practices  for  decision-making  or  problem  solving. 
While  some  checklists  were  used  for  procedures  such  as  troubleshooting  or  fault  isolation,  no 
other  job  aids  were  observed  to  assist  less  experienced  crewmembers.  No  models  were  observed 
for  organizing  information  for  decision-making  or  problem  solving  tasks.  Frequently,  SMEs 
commented  that  improved  decision-making  would  come  as  crewmembers  experienced  many 
different  situations  over  time.  However,  the  specifics  of  these  experiences  were  not  defined  nor 
were  any  of  the  specific  decision-making  skills  they  hoped  would  emerge  over  time. 

Gunnery  certification  training,  including  SPEAR  assessments,  are  often  based  on 
complex  scenarios  and  conditions.  However,  these  are  procedural  evaluations.  Scenarios 
providing  true  decision  situations  seldom  occurred.  When  crewmembers  performed  poorly,  the 
same  scenario  or  close  variations  of  it  were  rerun.  Crewmembers  tended  to  reapply  the 
procedural  steps  they  used  previously  to  address  issues  instead  of  applying  problem  solving 
skills  to  arrive  at  novel  solutions.  The  end  goal  was  doing  it  right,  not  necessarily  understanding 
why  it  was  right  or  transferring  what  was  learned  to  other  situations. 

In  summary,  Phase  I  analysis  indicated  that: 

•  Crewmember  (TCO  and  TCA)  training  and  preparations  are  focused  on  lower-order 
thinking  skills,  primarily  those  that  supported  the  correct  performance  of  routine  and 
critical  procedures. 

•  The  development  and  improvement  of  decision-making  skills  in  crewmembers  could 
benefit  from  the  compressed  practice  of  decision-making  skills  related  to  crew  duties. 
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•  Beneficial  practice  must  include  feedback  to  the  practicing  crewmember  and  should 
provide  some  form  of  feedback  to  unit  trainers  and  leaders. 

•  Decision  making  in  the  ECS  could  benefit  from  some  structure  or  organization  to  help  the 
crewmember  visualize  the  problem  and  its  context,  and  examine  relevant  information. 
However,  the  confined  space  of  the  work  area  and  the  rapidly  changing  situation  are  not 
conducive  to  a  chart-type  job  aid  and  there  is  no  time  to  record  data. 

•  Any  training  developed  must  complement  and  not  supplement  engagement  skills  and 
procedural  training. 

•  To  avoid  classification  restrictions  while  providing  authenticity,  training  would  need  to 
reflect  a  fictional  but  plausible  situation.  A  realistic  but  unclassified  enemy  air  and 
missile  order  of  battle  would  need  to  be  developed  and  some  unclassified  technical 
information  on  systems  made  available  for  use  in  training. 

•  The  user/training  audience  should  be  cast  in  a  familiar  role. 

Our  analysis  also  yielded  some  additional  questions: 

•  What  realistic  situations  could  crewmembers  encounter  that  would  elicit  decision-making 
skills? 

•  What  feedback  scheme  could  be  used  to  support  learning?  SMEs  sometimes  disagree  on 
the  correct  solutions. 

•  What  potential  models  can  be  applied  to  support  decision-making  across  a  broad  set  of 
problems  in  the  military  context? 

•  What  instructional  designs  and  principles  would  be  most  applicable  to  Patriot  ECS 
crewmembers?  The  crewmember/user  would  need  to  be  cast  in  a  role,  but  success  in  that 
role  needed  to  be  based  on  exercising  thinking  skills,  not  procedural  skills. 

Phase  II:  Defining  the  Training  Solution 


In  Phase  II,  the  research  team  defined  what  was  required  to  develop  a  training  prototype 
to  address  the  identified  shortcomings  in  decision-making  training.  The  ADDIE  process  model 
was  used  to  frame  its  development,  with  the  Analysis  step  applying  our  findings  from  Phase  I  of 
this  research. 

Phase  II  Method 

The  first  question  we  sought  to  address  concerned  what  situations  would  most  effectively 
elicit  decision-making  and  problem  solving  skills  for  ECS  crewmembers.  In  addition,  we  sought 
out  common  elements  of  those  situations  to  identify  a  job  aid  to  support  less  experienced 
crewmembers  in  their  decision-making  process. 

Decision- Triggering  Events 

The  research  team  spoke  with  26  SMEs  at  the  ADA  School  to  identify  situations  in 
Patriot  ECS  operations  that  elicit  complex  decision-making  skills.  The  following  describes  the 
background  experience  of  the  SMEs: 
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•  An  SSG  (14T30)  with  ten  years  in  Patriot  operational  system  repairs.  This  SME  had 
extensive  experience  leading  or  working  as  a  member  of  a  hot  crew.  A  hot-crew  is 
composed  of  an  NCOIC  and  one  or  more  additional  launcher  crewmembers  on-call  and 
on-hand  as  a  contact  team  to  rapidly  repair  or  refuel  Patriot  launching  stations. 

•  A  Top  Gun  Course  Manager  who  was  a  senior  warrant  officer  with  enlisted  time  as  a 
TCA,  Battery  Trainer,  battery  and  battalion  Master  Gunner,  TDA  in  the  ICC.  As  a 
Warrant  Officer,  he  had  extensive  experience  as  a  TCO  and  as  a  battalion  and  brigade 
Electronic  Missile  Maintenance  Officer/Standardization  Officer. 

•  Four  new  Warrant  Officers  (140E),  who  were  about  to  graduate  from  the  MOS  basic 
course.  Each  had  six  to  1 1  years  enlisted  experience.  Most  were  Patriot  Master  Gunner 
graduates  and  all  had  extensive  time  as  TCAs. 

•  Two  DA  Civilians  employed  by  the  ADA  School  as  Reconfigurable  Table  Top  Trainer 
(RT3)  trainers  and  training  developers.  One  was  a  former  Avenger  gunner.  However,  he 
had  extensive  knowledge  of  the  RT3,  RT3  software,  and  Patriot  ECS  operations.  The 
second  had  extensive  enlisted/NCO  experience  as  a  TCA  and  had  been  an  enlisted  TCO 
prior  to  leaving  active  duty  and  assuming  a  GS  position.  These  technicians  maintained 
RT3  software  and  were  retained  by  the  School  to  develop  customized  training  packages 
for  in-unit  training. 

•  Two  CW4s  assigned  to  the  Warrant  Officer  Basic  Course  (WOBC)  as  the  Course 
Manager  and  as  an  Instructor/Writer.  Both  had  extensive  enlisted  experience  on  Patriot 
and  they  had  both  served  as  TCOs  in  the  ECS,  Tactical  Directors  in  the  ICC,  and  as 
battalion  and  brigade  Electronic  Missile  Maintenance  Officers/Standardization  Officers. 

•  Five  CW2s  in  the  final  days  of  their  WO  Advanced  Course.  The  group  was  mixed  with 
experience  concerning  both  Patriot  maintenance  and  operations.  They  represented 
extensive  enlisted  experience  (five-and-a-half  to  10  years).  All  had  tactical/operations 
experience. 

•  Eleven  SMEs  composed  of  nine  Master  Gunner  students  and  two  Master  Gunner  Course 
Trainers.  While  their  actual  time  in  operations  and  engagement  crew  duties  varied,  all 
students  were  approaching  graduation  from  the  Patriot  Master  Gunner  course.  Seven 
expected  to  return  immediately  to  ECS  or  trainer  duties  in  a  Patriot  battery.  One  was 
slated  to  become  a  Platoon  Sergeant  and  one  was  a  battery  First  Sergeant  designee. 

From  these  discussions,  we  identified  nine  situations  or  elements  of  situations  that  elicit 
decision-making  skills.  These  situations  incorporated  plausible  tactical  problems,  maintenance 
issues,  and  unexpected  circumstances — the  raw  materials  for  realistic  scenarios  and  problem 
situations.  While  some  events  were  only  discussed  or  put  forward  by  a  single  source,  their 
plausibility  and  validity  was  verified  across  multiple  sources.  The  uniqueness  of  some 
situations,  the  frequency  with  which  others  were  mentioned,  and  the  circumstances  of  the 
accounts  indicated  that  these  situations  could  be  developed  into  training  scenarios  requiring 
higher-order  thinking  and/or  problem  solving  skills  on  the  part  of  ECS  crewmembers.  Table  3 
describes  the  decision-triggering  situations  we  derived  based  on  our  discussions  with  SMEs. 
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Table  3 


Decision-Triggering  Situations  Described  by  Patriot  Subject  Matter  Experts 


Decision-Triggering  Situations 

Conditions/Examples 

Determining  the  mission  impact  of  a  fault 

The  system  capabilities  are  degraded  but  the  Patriot  can 
still  engage  -  degraded  operations;  launcher  problems; 
recommend  repair  priorities/  modify  operations  as 
required. 

Recommending  reload  and  maintenance 
priorities 

Time  and  task  management  -  during  air  battle;  fault  ID; 
launcher  and  system  maintenance;  reloading. 

Reacting  to  an  ARM  launch 

Saturate  the  environment  (friendly  and  enemy);  screening 
ARM  carriers;  alert  line/screen  warnings;  misclassified 
missile  track  or  type. 

Executing  the  directed  engagement  of  a  track 

Merge  tracks  for  enemy  and  friendly  aircraft  (hazard); 
depict  threat  systems  in  a  friendly  air  corridor  at  a 
protected  altitude. 

Acting  to  clarify  an  erroneous  or  misclassification 

TBMs/ARMs/ASMs;  slow  ABTs  or  TBMs;  bad  track  or 
tab  data. 

Executing  cruise  missile  (air  or  land  launched) 

Portray  a  high,  fast  track  or  a  high  launch  and  descent  out 

engagement 

of  radar  coverage. 

Executing  engagements  of  threatening  massed 
ABTs 

Varied  aircraft  in  coordinated  effort.  Determine  the 
highest  threat;  multiple  ABTs  emerge  from  single  track. 

Acting  against  swarm  attacks  or  massed 
suppression  of  enemy  defenses  (SEAD) 

Swarm;  multiple  ARM  launches. 

Executing  engagement  of  varied  types  of  TBMs 
and  mixed  multiple  threats 

Mix  TBMs;  saturate  with  coordinated  ABT  and  ARM 
attacks 

Organizing  Data  Elements 

Research  has  shown  that  graphic  organizers  can  be  a  successful  tool  for  teaching  thinking 
skills  (Bums  et  al.,  2006).  Graphic  organizers  provide  structure  and  organization  as  students 
develop  their  thinking  and  problem  solving  skills.  Seven  discrete  thinking  skills  have  been 
taught  effectively  using  graphic  organizers,  including: 

•  Determining  cause  and  effect, 

•  Making  decisions, 

•  Comparing  and  contrasting  information, 

•  Classifying  information, 

•  Making  observations, 

•  Planning  for  future  events,  and 

•  Predicting  outcomes. 

These  thinking  skills  are  applicable  to  the  duties  and  actions  of  Patriot  ECS  crews  during 
air  battle  operations.  This  suggests  a  graphic  organizer  could  benefit  the  ECS  crewmembers  in 
developing  their  critical  thinking  and  decision-making  skills.  The  limitations  of  using  a 
traditional  graphic  organizer  in  the  context  of  Patriot  are: 

•  Confined  space  in  the  van  and  at  the  ECS  workstation, 
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•  Rapid  pace  of  operations  in  an  air  battle, 

•  Frequent  changes  and  updates  to  data  and  information  available, 

•  High  probability  that  inputs  for  multiple  decisions  are  constantly  being  assimilated, 

•  Available  information  relates  to  more  than  a  single  decision  or  action,  and 

•  Actions  related  to  some  decisions  may  be  interrelated  with  another  decision. 

We  developed  a  simple  graphic  organizer  to  serve  as  a  conceptual  framework  to  assist  Soldiers 
in  identifying  factors  that  may  constrain  their  decision-making.  This  model  is  utilized 
throughout  the  training  scenarios  the  research  team  developed.  When  engaged  in  Patriot 
operations,  ECS  crewmembers  focus  on  their  radar  scopes,  supplemented  by  information  derived 
from  tabs,  tables,  alert  lines,  and  brief  liner  displays.  Crewmembers  may  attend  to  many 
different  pieces  of  information  at  a  given  time.  The  graphic  organizer  needed  to  facilitate  a 
crewmember  in  quickly  attending  to  pertinent  information  in  the  context  of  the  total  situation. 
Likewise,  the  organizer  needed  to  alert  crewmembers  to  missing  information,  things  they  may 
have  overlooked.  We  intended  that  the  graphic  organizer  allow  crewmembers  to  distinguish 
relevant  attributes  of  available  information,  and  evaluate,  assess,  and  compare  information  in 
determining  what  actions  are  required.  The  graphic  organizer  would  need  to  be  both  reactive  and 
proactive — respectively  providing  support  leading  to  a  decision  and  anticipating  future  decisions 
in  response  to  enemy  intentions  and  actions. 

Building  a  Database  for  Decision  Related  Information 

The  requirements  for  an  instructional  design  model  were  determined  and  draft  scenarios 
were  developed.  The  scenarios  needed  to  be  set  in  the  context  of  an  unclassified  general 
situation,  not  associated  with  an  existing  mission  or  contingency  area.  The  research  team 
developed  an  unclassified  air  and  missile  order  of  battle  to  present  a  capable  threat.  We  used 
dated  materials  (e.g.,  Department  of  the  Army,  1991)  to  derive  examples  of  threat  air  force  and 
missile  elements.  Unclassified,  but  credible,  websites  such  as  MISSILE  THREAT  provided 
detailed  information  on  threat  ballistic  missile  systems.8  IHS,  Inc. — the  holder  of  Jane’s 
Information  Group — provided  an  unclassified,  open-source  intelligence  database  of  defense  and 
security  information,  frequently  used  by  U.S.  and  NATO  intelligence  communities.  Using  threat 
systems  derived  from  the  Cold  War  era  provided  abundant  technical  information  and  capabilities 
on  long-exploited  systems  still  in  service  in  some  nations. 

A  fictional  area  of  operation  and  general  situation  were  developed  using  the  southeastern 
United  States.  A  complete  Air  and  Missile  Defense  annex  was  created  for  the  fictional  Joint 
Task  Force  (JTF),  deploying  with  an  AMD  Task  Force  (TF).  The  JTF  deployment  was  an  action 
in  support  of  a  developing  friendly  nation  being  attacked  by  a  more  technologically  advanced, 
well-armed,  and  hostile  neighboring  nation.  A  special  situation  was  developed,  providing 
defensive  operations  by  a  Patriot  battery  positioned  to  protect  the  forward  assembly  area  of  a 
major  ground  component  of  the  JTF.  The  research  team  was  careful  to  draw  all  information 
from  open  sources,  and  to  keep  all  materials  unrelated  to  possible  contingencies  or  real-world 
events. 


8  http://missilethreat.com/missile-class/tactical-ballistic-missile-tbm/.  MISSILE  THREAT  is  a  project  of  the  George 
C.  Marshall  and  Claremont  Institutes. 
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When  designing  the  PCST,  the  research  team  determined  that  it  should  focus  on 
developing  critical  thinking  and  decision-making  skills  over  procedural  and  technical  skills 
specific  to  the  Patriot  system.  As  such,  it  would  only  require  the  most  basic  knowledge  of  the 
actual  Patriot  system.  Users  would  need  to  be  able  to  read  the  symbols  and  track  icons  generated 
on  scope  in  the  ECS.  Users  would  not  need  to  know  specific  switches,  controls,  indicators,  and 
tabs  of  the  Patriot  system.  The  PCST  would  not  reference  or  use  any  classified  information  and 
would  not  require  detailed  understanding  or  divulging  of  sensitive  embedded  resources  or 
processing  capabilities  of  the  Patriot  system. 

Scope  views  were  presented  as  a  series  of  progressive  still  images,  rather  than  a  full 
motion  displays.  Users  were  not  required  to  manipulate  the  switches,  controls,  indicators,  and 
tabs  of  the  Patriot  to  compile  information  from  multiple  embedded  sources.  Data  would  be 
presented  already  compiled.  These  simplified  data  presentations  were  created  for  each  step  in  a 
scenario.  This  would  allow  the  PCST  to  focus  on  developing  data  interpretation  and  decision¬ 
making  skills  rather  than  the  procedural  skills  related  to  compiling  data  during  normal  Patriot 
operations. 

Users  would  be  provided  feedback  on  how  they  performed  on  each  scenario.  A  number 
of  assessment  schemes  were  considered.  The  research  team  determined  that  the  performance  of 
the  user  should  be  compared  against  that  of  an  experienced  Patriot  expert.  In  this  scheme,  the 
user  selects  a  decision  set  and  then  identifies  the  factors  that  most  influenced  the  ranking, 
priority,  or  selection  of  the  decision  set.  We  scored  users  against  the  consensus  responses  of  a 
group  of  senior  Patriot  SMEs.  Scoring  user  selections  in  this  way  moved  the  assessment  away 
from  absolutes  and  toward  a  visual  and  auditory  feedback  system  that  rated  how  close  users’ 
individual  selections  came  to  those  of  the  SME  consensus. 

Materials  for  the  PCST  were  reviewed  by  experts  from  32nd  Air  and  Missile  Defense 
Command’s  Directorate  of  Plans,  Training,  Mobilization,  and  Security  (i.e.,  G3)  staff  and 
SPEAR  Team.  Based  these  reviews,  the  research  team  further  refined  the  PCST  design. 

Working  sessions  were  conducted  with  these  experts  to  mature  further  the  draft  scenarios,  data, 
presentations,  and  assessment  schema.  Both  active  duty  military  and  contractors  contributed  to 
these  sessions,  making  corrections  and  adjustments  to  the  problem  sets  and  associated  scenarios, 
how  data  were  presented,  and  the  assessment  and  user  feedback  scheme. 

The  tool  was  organized  in  terms  of  an  introduction  followed  by  five  training  modules. 
Each  module  presented  a  realistic  scenario  and  contained  decision-triggering  situations 
recommended  by  the  ADA  School  SMEs.  The  scenarios  and  decision-triggers  were  intended  to 
elicit  and  exercise  aspects  of  critical  thinking.  The  design  incorporated  one  basic,  one 
intermediate,  and  three  advanced  or  mission  modules.  The  degree  of  scaffolding  support 
provided  to  users  was  tapered  and  the  number  of  potential  decision-triggers  was  increased  as  the 
training  moved  to  more  advanced  levels.  Early  in  training,  users  received  feedback  on  each 
decision;  later  in  training,  multiple  decisions  were  made  before  feedback.  Each  module  included 
an  AAR  that  scored  the  users’  responses  against  Patriot  SME  responses.  At  the  end  of  the 
modules  for  a  section,  users  were  provided  with  overall  feedback  on  how  their  performance 
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compared  to  that  of  experts.  Users  were  able  to  print  this  page  to  provide  to  unit  trainers  or 
supervisors. 

The  storyboards  for  the  five  training  modules  were  prepared  and  presented  to  Patriot 
SMEs  at  the  Air  Defense  School.  The  SMEs  reviewed  the  storyboards  for  accuracy  and 
authenticity,  and  to  provide  their  input  to  the  consensus  decisions  used  as  the  foundation  for  the 
performance  feedback  system.  This  panel  consisted  of: 

•  Patriot  Top  Gun  Course  Manager  -  CW3  with  almost  five  years  of  experience  as  a  TCA, 
two  years  of  experience  as  a  TCO,  and  over  four  years  of  experience  as  a  Standardization 
Officer. 

•  Patriot  Warrant  Officer  Basic  Course  (140E)  Course  Instructor/Writer  -  CW4  with  two 
years  of  experience  as  a  TCO,  two  years  of  experience  as  the  ICC  TD  spanning  two 
battalions,  and  two-and-a-half  years  of  experience  as  a  Standardization  Officer  spanning 
two  battalions. 

•  A  recent  graduate  of  the  Warrant  Officer  Basic  Course  (140E)  -  WOl  with  six  months 
experience  as  a  TCA  and  two  years  of  experience  as  a  TDA  in  a  battalion  ICC. 

•  Patriot  Master  Gunner  Course  Instructor/Writer  -  SSG  with  almost  two  years  as  a  TCA, 
over  one  year  as  a  platoon  sergeant,  and  one  year  as  a  battery  Master  Gunner  and  trainer. 

In  order  to  derive  consensus  decisions  based  on  the  scenarios  presented,  the  research 
team  conducted  extended  dialog  with  participants.  These  discussions  concerned  the  rationales 
involved  in  the  tactical  and  other  supporting  factors  that  contributed  to  the  SMEs’  decision¬ 
making  processes.  The  SMEs  viewed  all  storyboards,  including  images  and  data  files.  They 
reviewed  and  discussed  all  decision  and  supporting  data.  Their  consensus  was  documented  to 
provide  a  basis  for  assessing  users  of  the  PCST.  After  all  final  input  and  critical  reviews  were 
incorporated  into  the  storyboards  and  accompanying  materials,  they  were  turned  over  to  media 
developers  to  produce  the  training  product. 

Results  of  the  Phase  II  Analysis 

With  a  focus  on  the  process  of  developing  the  PCST  for  Patriot  ECS,  the  following 
results  address  three  key  issues: 

•  Defining  the  deficiencies  in  existing  Patriot  individual  and  ECS  crews  training  that 
perpetuate  shortcomings  in  critical  thinking  skills, 

•  Identifying  and  selecting  training  techniques  and  methods  that  could  be  employed  in  a 
multi-media  tool  to  address  these  deficiencies  and  build  them  into  a  practical  prototype 
PCST,  and 

•  Validating  the  prototype  PCST. 

The  PCST  was  developed  using  the  ADDIE  process,  augmented  by  principles  derived 
from  research  addressing  multimedia  instructional  design  (Blankenbeckler,  Graves,  &  Wampler, 
2014;  Blankenbeckler,  Graves,  Dlubac,  &  Wampler,  2016;  Ingurgio,  Blankenbeckler,  & 
Wampler,  in  press)  and  more  general  instructional  principles  (Merrill,  2002). 
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Common  Elements  of  Decision-Triggering  Situations 

When  the  research  team  analyzed  the  decision-triggering  situations  in  greater  depth,  data 
elements  that  were  common  to  all  the  situations  were  identified.  While  the  salience  of  elements 
varied  across  different  situations,  a  general  process  for  considering  the  elements  arose  from  this 
analysis,  which  pointed  the  research  team  to  a  common  process  for  identifying  and  evaluating 
the  elements  in  specific  decision-triggering  situations.  This  model  provided  a  basis  for  a  graphic 
organizer  to  support  crewmember  analysis,  problem-solving,  and  decision-making. 

The  core  element  of  every  decision  is  how  it  will  affect  the  units’  ability  to  sustain  its 
mission.  To  determine  this,  a  decision-maker  must  consider  some  aspect  of  the  enemy  situation 
and  the  enemy’s  ability  to  react  to  the  decision.  The  battlespace  is  key  in  this  determination, 
including  the  air  and  terrain  situation  in  the  units’  area  of  operation.  In  addition,  the  decision¬ 
maker  needs  to  consider  the  origin  of  hostile  aircraft  and  missiles,  their  current  locations,  transit 
times  and  airspeeds,  launcher  locations,  weapon  types  and  warhead  capabilities.  Decision¬ 
makers  also  need  to  consider  the  situation  from  the  standpoint  of  their  own  systems.  For 
instance,  what  the  equipment  and  missile/ammunition  status  of  the  Patriot  system  is,  as  well  as 
what  maintenance  and  repair  is  needed.  Time  to  repair,  time  to  reload,  time  of  flight,  last  time  to 
launch,  time  to  impact — all  were  potential  factors  adding  an  element  of  urgency  to  the  decision¬ 
making  process. 

Members  of  the  research  team  with  prior  Army  service  recognized  that  the  METT-TC 
process  could  serve  to  organize  the  information  required  to  come  to  a  decision  in  response  to  a 
decision-triggering  situation.  METT-TC  is  acronym  for  the  factors  fundamental  to  assessing, 
estimating,  and  visualizing  a  tactical  situation:  Mission,  Enemy,  Terrain  and  weather,  Troops, 
equipment,  and  support  available.  Time  available,  and  Civil  considerations  (Department  of  the 
Army,  2008a).  Rather  than  repackage  the  process  as  something  new,  we  applied  the  familiar 
Army  process  to  the  Patriot  ECS  decision-making  context.  Civil  considerations  were  not  a 
distinct  factor  in  our  discussions  with  Patriot  SMEs.  As  a  result,  we  organized  the  evaluation 
process  around  METT-T. 

The  research  team  developed  draft  scenarios  and  an  interface  providing  data  emulating 
the  Patriot  ECS.  While  data  were  presented  in  a  manner  similar  to  the  ECS,  the  arrangement 
presented  situations  for  a  decision,  permitting  the  user  to  select  an  action  and  to  prioritize  the 
information  that  influenced  their  decision.  The  scenarios  presented  were  unclassified,  based  on  a 
general  mission  in  an  area  of  operation  that  was  in  the  southeastern  United  States.  A  fictional 
belligerent  nation  had  forcibly  annexed  a  border  province  of  a  U.S.  ally.  Threat  and  situational 
information  drawn  from  open  sources  was  consolidated  into  an  Air  and  Missile  Defense 
Appendix  to  the  Fires  Annex  of  a  fictional  Joint  Task  Force  Operations  Order.  This  converted 
the  information  to  a  form  expected  by  the  tool  user.  Again,  the  research  team  was  careful  to 
draw  all  information  from  open  sources  and  to  keep  all  scenarios  unrelated  to  possible  real-world 
contingencies,  events,  or  the  classified  capabilities  and  characteristics  of  air  and  missile 
platforms. 

The  design  of  the  tool  was  such  that  use  would  require  only  the  most  basic  knowledge  of 
the  actual  Patriot  system.  Snapshot  images  of  the  scope  were  developed  to  support  the  scenarios, 
replacing  the  constantly  changing  radar  images.  These  sequentially  changing  images  provide 
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users  with  selected  views,  similar  to  those  routinely  available  through  the  scope  displays  of  the 
ECS  van.  Sequential  images  and  brief  text  or  narrative  updates  take  the  place  of  the  constantly 
changing  real-time  radar  images  on  the  TCO  and  TCA  scopes. 

Previous  research  identified  and  documented  instructional  design  principles  to  enhance 
the  learning  effectiveness  of  interactive  multi-media  instruction  products  (Blankenbeckler, 
Graves,  &  Wampler,  2013,  2014;  Graves,  Blankenbeckler,  Wampler,  &  Roberts,  2016; 
Blankenbeckler  et  al.,  2016;  Ingurgio  et  al.,  in  press).  We  reviewed  the  principles  and  drafted  an 
instructional  design  concept.  The  design  concept  focused  on  the  principles  and  generalizable 
strategies  supported  by  relevant  research  literature,  expert  knowledge,  and  practical  experience. 
The  principles  are  summarized  below: 

Define  the  learning  task  -  what  the  learner  should  know  or  be  able  to  do  after 
instruction.  The  learning  task  for  this  effort  is  skill  focused.  The  learning  goal  is  to  have  the 
new  learners  make  and  select  decisions  similar  to  those  expressed  in  the  consensus  of  Patriot 
SMEs.  A  secondary  goal  of  training  is  to  model  the  users’  decision  processes.  Not  only  do  we 
assess  the  decision,  but  by  ranking  the  data  elements  that  most  prompted  the  decision  selections, 
the  Tool  leads  the  user  toward  a  more  refined  practice  of  decision-making. 

Link  new  knowledge  and  skills  to  real  world  tasks  and  missions  in  context  (problem 
based).  Real  world  problems  in  a  contingency  mission  context  are  the  focus  of  the  training. 
While  the  situations  and  enemy  order  of  battle  are  fictitious,  equipment  capabilities  and  threat 
tactics  are  real.  Users  are  given  actual  problems  to  solve  that  impact  their  ability  to  protect 
elements  of  the  ground  component  of  a  U.S.  JTF.  The  displays  and  data  provided  are  the  same 
as  would  be  expected  from  the  Patriot  system.  The  content  is  chunked  into  small  sequential 
problems  and  the  user  is  provided  with  related  data  sets  arranged  in  a  graphic  organizer,  an 
appropriate  cognitive  representation  for  ease  of  understanding,  assimilation,  and  to  support  the 
management  of  essential  processing. 

Encourage  practice  of  new  knowledge  and  skills  in  context.  The  tool  design  provides 
for  user  decisions  in  varied  air  battle  situations.  It  provides  training  in  progressively  difficult  and 
complex  situations,  divided  into  introductory  or  crawl  training  (one  module);  intermediate  or 
walk  training  (one  module);  and  advanced  or  run  training  in  three  mission  modules.  There  are 
five  modules  in  total.  In  the  later  modules  instructional  scaffolding  is  diminished  as  learners’ 
facility  with  AMD  engagement  decision-making  processes  increases.  It  provides  for  both 
general  performance  feedback  at  the  end  of  the  module  as  well  as  specific  feedback  at  selected 
decision  points  (see  Appendix  B  for  example  scenarios). 

The  first  instructional  module  is  the  crawl  phase.  It  provides  three  decision  points  in  an 
evolving  situation.  While  working  through  the  module,  users  make  three  decisions  and  rate  the 
factors  that  most  contributed  to  their  decisions.  Then,  their  responses  are  compared  to  those  of 
senior  Patriot  SMEs.  An  SME  verbal  rationale  is  provided  for  each  decision  point.  At  the  end  of 
the  module,  an  AAR  compares  the  users’  decisions  to  those  of  the  SMEs. 

The  second  module  is  the  walk  phase,  providing  four  decision  points.  Users  are  provided 
scaffolding  in  terms  of  favorable  or  unfavorable  comparisons  to  SMEs.  If  they  have  a  low  score, 
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they  are  provided  an  opportunity  to  reconsider  their  decisions.  Again,  at  the  end  of  the  module, 
users  are  provided  an  AAR  comparing  their  decisions  and  selections  to  those  of  the  SMEs. 

The  final  three  modules  comprise  the  run  phase.  In  the  run  phase,  users  are  exposed  to 
scenarios  with  more  decision-triggers  than  the  first  and  second  modules.  In  addition,  the  third 
module  extends  the  number  of  decisions  users  must  make  before  receiving  direct  feedback. 
Instead,  feedback  comes  by  way  of  navigation  routing.  They  are  able  to  see  the  consequences  of 
their  decisions  and  how  their  decisions  are  influencing  their  later  options.  When  the  user  selects 
decisions  that  deviate  significantly  from  SME  selections,  they  are  routed  toward  catastrophic 
consequences.  They  are  then  allowed  to  reconsider  their  decisions.  At  the  end  of  the  run  phase 
module,  users  are  provided  with  feedback  that  compares  their  decisions  to  those  of  SMEs. 

Within  scenarios,  problems  were  introduced  visually  using  graphics  similar  to  those  of 
the  scopes  in  an  ECS.  Minor  displacements  of  track  markers  and  objects  from  frame  to  frame 
substituted  for  real-time  movements.  Movements  were  depicted  in  a  manner  to  show  the  relative 
speeds  of  track  icons.  For  example,  high-performance  aircraft  advanced  across  or  down  the 
scope  faster  than  UAVs.  TBMs  and  ARMs  were  depicted  as  moving  at  very  rapid  speeds. 
Standard  symbology  was  used  to  portray  each  track.  However,  no  time  references  were  provided 
on  these  frames.  For  identification,  all  tracks  are  labeled  with  a  discrete  reference  number  that 
remained  consistent  for  the  track  throughout  the  scenario.  Figures  4  and  5  below  provide 
samples  of  how  scenarios  progressed  from  frame  to  frame. 


Figure  4.  Scope  View  1 :  Radar  Image 
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In  the  Patriot  system,  data  on  tracks  or  events  is  accessed  through  multiple  switches, 
controls,  indicators,  and  tabs.  In  the  PCST,  data  was  presented  on  a  single  page  (see  Figure  6). 
Users  were  not  required  to  obtain  data  by  manipulating  the  system  and  then  compile  it  from 
multiple  sources.  Simplified  data  sets  were  created  for  each  element  in  a  scenario.  The  data  was 
displayed  to  according  to  the  scheme  of  the  graphic  organizer,  METT-T,  described  previously. 


METT-T  Data:  SCUD  and  ARM  Engagements 


WEAPONS  HOLD 


System  Status'  Amber 


Tracks  143  {JAMMER),  144.  and  123  (ACR)  are  classified 
as  Hostia  ARTS 
FF;  No  response 
Track  Data: 

Headng  Southeast  (170°)  toward  the  protecled  asset 
Attitude:  49.00  kft/  14.94  km.  steady 
Speed,  1,040  kt/ 540  m/s 
Point  of  origin,:  Bug 
:  17.3  n 


Track  145  is  dassrfted  as  an  ARkVASM 
FF  No  Response 
Track  Data; 


Launcher/Missile  Status  (PAC2  8/PAC3  22) 

Alert  LSNAAN  HAZARD 

1  Groon 

2  Groon 

4  Groon 

FP  Status  Panel  Launch  3  Indicators: 

PAC3X  10 

RAC2  X  4 

PAC3  X  12 

PAC2  X  4 

OPR/STBY: 

OPER: 

FUEL: 

DOL 

Oper/Groen  | 

Green 

Green 

Attitude,  47,30  kft/  14,42  km,  steep  dive  angle 
Speed:  1 ,600  kt/  930  rrVs 
Point  of  origin.  Track  129 
Length:  4.7  meters 

Track  1 13  is  dassrfed  as  Hostle  ART 
FF:  No  Response 
Track  Data: 

Headng  South  Southeast  (195°)  toward  the 
I  asset 

kft/ 1  28  km,  steady 
Speed:  120  kt/  62  m/s 
Point  of  origin  UNK 


Figure  6.  METT-T  factors  relating  to  a  decision. 
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The  example  in  Figure  6  includes  a  maintenance  fault.  On  the  METT-T  screens,  users 
were  given  an  option  to  return  to  the  previous  screens  to  review  scope  images.  Launcher  status 
and  missile  counts  were  also  provided.  Given  that  users  come  from  units  with  mission  areas  that 
span  the  globe,  metric  and  imperial  measures  were  reported.  Radar-measured  lengths  were 
provided  in  meters. 

Assessment  and  Scoring 

The  research  team  developed  a  scheme  to  assess  the  performance  of  users.  This  scheme 
used  the  consensus  solution  of  Patriot  experts  as  the  criterion.  The  assessment  asked  users  to 
make  decisions  then  identify  factors  that  influenced  their  decisions.  User  selections  were  scored 
in  terms  of  how  close  they  were  to  Patriot  SME  selections.  While  time  is  a  factor  in  making 
decisions,  the  tool  was  focused  on  proficiency  over  speed.  Given  a  data  set  and  a  scenario,  can 
the  user  make  decisions  that  approximate  those  of  Patriot  experts?  Users’  responses  were  scored 
as  being  dissimilar,  related  to,  or  similar  to  the  SMEs’  consensus  response. 

In  establishing  the  criterion  for  scoring,  we  worked  with  a  panel  of  Patriot  SMEs.  The 
comments,  decisions,  and  inputs  of  the  Patriot  SME  panel  varied  for  each  scenario.  Individual 
experiences  weighed  heavily  in  these  discussions.  Rather  than  establishing  an  absolute  best 
answer,  the  research  team  decided  that  users  should  be  scored  in  terms  of  how  closely  they 
approximated  the  SMEs’  consensus  solution. 

Scoring  considered  two  aspects  of  users’  performance.  First,  it  focused  on  the  decision(s) 
they  selected.  Second,  it  focused  on  the  factors  that  influenced  their  decision(s).  Most  often, 
users  were  asked  either  to  make  a  decision  or  to  prioritize,  from  a  list,  three  choices  of  how  they 
would  respond  to  a  situation.  Prioritized  responses  were  scored  in  terms  of  how  closely  they 
reflected  the  sequence  selected  by  the  SMEs.  The  closer  a  response  reflected  the  sequence 
selected  by  SMEs,  the  better  the  score.  Options  that  were  not  prioritized  by  the  SMEs  (i.e., 
distractor  items)  were  all  treated  equally  in  the  scoring  system. 

When  user  responses  are  scored,  the  tool  calculates  the  variation  of  user  responses  from 
SME  responses  using  the  explained  sum  of  squares  (ESS)  model  of  linear  regression.  Users’ 
scores  are  calculated  by  adding  up  the  squared  differences  between  the  users’  responses  and  the 
SMEs’  responses.  The  differences  were  squared  only  for  mathematical  purposes:  so  a  negative 
number  resulting  from  subtracting  users’  responses  from  SMEs’  responses  would  be  made 
positive  before  adding  the  differences  together  for  a  deviation  score.  If  differences  were  not 
squared,  extreme  user  responding  on  either  side  of  the  SMEs’  response  line  would  cancel  out. 
The  research  team  treated  all  SME  responses  as  correct,  so  the  ESS  generated  from  the  SMEs’ 
responses  to  correct  user  responses  is  linear,  with  a  deviation  of  zero.  The  more  users’  responses 
deviated  from  SMEs’  responses,  the  larger  the  number  representing  the  difference.  Therefore,  a 
higher  sum  of  squared  deviations  for  a  given  user  represents  less  similar  responding  when  they 
are  compared  to  the  SMEs.  Likewise,  the  closer  to  zero  the  sum  of  squared  deviations  was  for  a 
given  user,  the  more  similar  their  responding  was  to  the  SMEs. 

Consider  the  example  data  presented  in  Table  4.  The  example  shows  SME  responses  as 
baseline,  with  a  deviation  score  of  zero,  and  three  sample  user  responses  as  well  as  the 
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calculations  of  the  ESS  deviation  score  for  each.  In  addition,  recall  the  distractor  response  was 
defined  in  terms  of  no  SME  endorsing  the  response.  If  a  hypothetical  user  were  to  respond  with 
all  distractor  options,  then  they  would  earn  the  worst  possible  score,  a  score  indicating  the 
highest  degree  of  deviation  from  the  SMEs.  In  this  example,  that  score  would  be  26.  Note  that 
SME  responses  are  a  linear  function.  In  the  example,  User  #l’s  decisions  (score  =  24)  are  less 
similar  to  SME  decisions  than  User  #3  (score  =  8).  User  #2  decisions  (score  =  2)  are  the  most 
similar  to  SME  decisions  among  the  three  users,  as  this  users’  score  is  the  closest  to  zero. 

Table  4 


Example  of  Prioritized  User  Decisions 


Options 

User  Decisions  (Prioritized) 

SME  User  #1  User  #2 

Baseline 

User  #3 

Distractor 

Response 

A 

1 

4 

1 

3 

4 

B 

4 

3 

4 

4 

2 

C 

2 

4 

2 

2 

4 

D 

4 

1 

3 

4 

1 

E 

3 

2 

4 

1 

3 

ESS  Scoring 

A 

(1  —  l)2  =  0 

(4  -  1)2=  9 

(1  -  l)2  =  0 

(3  —  l)2  =  4 

(4  —  l)2  =  9 

B 

(4  -  4)2  =  0 

(3  -4)2=  1 

O 

II 

Cl 

1 

(4  -  4)2  =  0 

(2  -  4)2  =  4 

C 

(2  -  2)2  =  0 

(4  -  2)2  =  4 

(2  -  2)2  =  0 

(2  -  2)2  =  0 

(4  -  2)2  =  4 

D 

(4  -  4)2  =  0 

(1  -  4)2  =  9 

(3  -4)2=  1 

(4  -  4)2  =  0 

(1  -  4)2  =  9 

E 

(3  -  3)2  =  0 

(2  -  3)2=  1 

(4  -  3)2  =  1 

(1  -3)2  =  4 

(3  -  3)2  =  0 

Sum  (ESS) 

0 

24 

2 

8 

26 

The  research  team  decided  that  to  provide  effective  feedback  the  deviation  scores  for 
users  would  need  to  be  translated  into  readily  interpretable  categories.  Users  were  given 
feedback  based  on  whether  their  scores  were  “similar,”  “related,”  or  “dissimilar”  to  those  of 
SMEs.  These  categories  were  determined  by  calculating  the  best  and  worst  possible  scores  for 
each  decision  point.  The  best  score  is  always  zero,  however,  the  worst  score  varied  depending 
on  the  number  of  response  options  available  for  each  scenario.  The  worst  possible  score  for  each 
scenario  was  summed  from  the  worst  possible  scores  for  each  decision  point  within  a  scenario — 
a  summed  score  which  would  indicate  that  the  user’s  responses  were  100%  dissimilar  from  the 
SMEs.  Categorical  scores  were  interpolated  to  percentages  from  the  raw  scores  calculated  for 
each  scenario.  If  a  user  deviated  from  the  SMEs  by  more  than  50%  they  were  rated  as 
dissimilar,  between  20%  and  49%  they  were  rated  as  related ;  and  19%  or  less,  they  were  rated  as 
similar  (see  Table  5).  Consider  a  hypothetical  scenario  that  has  a  worst  possible  score  of  82. 
Using  the  percentages  for  each  category,  ESS  scores  from  0  to  16  would  be  rated  as  similar, 
those  from  17  to  41  would  be  rated  as  related,  and  those  from  42  to  82  would  be  rated  as 
dissimilar. 
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Table  5 


Ranges  of  Percentages  used  for  User  Feedback 


Total  ESS  % 

Similar 

0  to  19% 

Related 

20%  to  49% 

Dissimilar 

50%  to  100% 

While  the  above  discussion  of  scoring  focused  on  prioritized  responses,  in  a  few 
instances  decision  points  within  a  scenario  ask  users  to  make  dichotomous  decisions.  For  those 
cases,  either  user  responses  match  the  SMEs’  or  they  do  not.  Others  decision  points  ask  users  to 
rate  the  likelihood  of  an  event  based  on  their  present  information.  In  both  cases,  similar  ESS 
calculations  were  made  for  scoring  purposes. 

In  addition  to  selecting/prioritizing  decisions,  users  were  also  asked  to  indicate  the  factors 
contributing  to  their  decisions.  These  factors  were  not  prioritized.  As  above,  these  scores  were 
calculated  in  terms  of  deviation  from  the  SME  consensus.  The  math  was  simplified,  however,  as 
it  did  not  need  to  account  for  the  order  in  which  responses  were  prioritized. 

The  last  page  of  each  section  of  the  PCST  presents  an  AAR,  providing  scoring  and 
auditory  feedback  to  the  user  concerning  the  SME  consensus  decision(s)  and  rationale  for  those 
decisions.  This  feedback  is  reduced  in  the  run  phase  of  training,  as  users  were  expected  to 
diagnose  where  they  had  made  decisions  that  produced  negative  outcomes.  The  users’ 
performance  is  compared  to  SMEs  in  terms  of  both  their  decisions  and  their  selection  of  factors 
contributing  to  their  decisions.  The  AAR  page  is  printable  and  could  be  provided  to  unit  trainers 
or  unit  chain-of-command  to  track  performance  and  progress.  Figure  7  provides  an  example  of 
an  end  of  module  AAR.  The  closed  caption  (CC)  narration  is  displayed  for  this  instructional 
frame. 
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Figure  7.  Example  Training  Module  AAR  Page 

Phase  III:  Validation  of  the  Patriot  Cognitive  Skills  Trainer 

In  the  third  phase  of  the  research,  we  validated  the  PCST  with  Patriot  operators  at  the 
108th  ADA  BDE,  Fort  Bragg,  NC.  Twenty  ( N  =  20)  Soldiers  gathered  in  a  classroom  at  a  BDE 
facility.  On  arrival,  we  briefed  the  Soldiers  on  the  research  purpose,  their  rights  as  participants, 
and  administered  informed  consent.  All  Soldiers  chose  to  participate.  Soldiers  were  seated  at 
desks,  allowing  sufficient  space  to  operate  laptop  computers  with  a  mouse  and  to  fill-out  the  data 
collection  forms.  Soldiers  were  provided  headphones  so  that  computer  audio  would  not  become 
a  distraction. 

Procedure 

Following  informed  consent,  Soldiers  completed  a  pre-training  questionnaire.  The  pre¬ 
training  questionnaire  asked  questions  about  demographics  (e.g.,  rank,  MOS),  background 
experience  and  training  related  to  their  Patriot  assignments.  They  then  completed  the  five 
training  modules.  After  completing  each  module,  they  recorded  their  progress  on  a  data 
collection  form.  Once  training  was  complete,  they  filled  out  a  post-training  questionnaire  and  a 
questionnaire  to  evaluate  the  training  design,  content,  and  their  learning  experience.  The  full 
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data  collection  session  required  two-and-a-half  to  three  hours.  The  average  time  spent  training 
was  two  hours,  13  minutes.9 

Participants 

Table  6  presents  the  sample  demographics.  The  sample  composition  matched  the 
intended  audience  for  the  trainer,  except  that  no  commissioned  officers — who  would  have  been 
serving  in  the  TCO  position — participated. 

Table  6 


Demographic  Characteristics  of  the  Sample 


Demographic  Variable 

Sample  Size 

N  =20 

Rank 

Private  First  Class  (PFC  E-3)  =  2  (10%) 

Specialist  (SPC  E-4)  =  13  (65%) 

Sergeant  (SGT  E-5)  =  1  (5%) 

Chief  Warrant  Officer  2  (CW2)  =  1  (5%) 

Chief  Warrant  Officer  3  (CW3)  =  3  (15%) 

MOS 

14E  =  16  (80%) 

HOE  =  4  (20%) 

Assignment 

Battery  Command  Post  =  1  (5%) 

Battalion  Readiness  Center  =  1  (5%) 

Communications  Operator  =  1  (5%) 

Standardization  Officer  =  2  (10%) 

Tactical  Control  Assistant  =13  (65%) 

Tactical  Director  =  1  (5%) 

Time  in  Current  Assignment 

M  =  14.6  mo. 

SD  =  16.5  mo. 

Range  =  1  to  61  mo. 

Distribution: 

1  to  6  mo.  =  1 1  (55%) 

15  to  24  mo.  =  4  (20%) 

30  to  61  mo.  =  4  (20%) 

Unknown  =  1  (5%) 

Highest  Gunnery  Table/RAL 
Certification 

II  (Ready-for-Action  Drills)  =  1  (5%) 

IV  (Basic  Gunnery  Certification)  =  1  (5%) 

VII  (Precertification  Tables  V  and  VI)  =  2  (10%) 

VIII  (Intermediate-Level  Gunnery  Certification)  =  9  (45%) 

XII  (Advanced-Level  Gunnery  Certification)  =  2  (10%) 

RAL11  =  3  (15%) 

RAL17  =  3  (15%) 

Q 

Crawl  phase  training  required  an  average  of  34  minutes.  Walk  phase  required  36  minutes.  The  average  for  the 
Run  phase  training  was  63  minutes. 
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Of  ./V  =  20  participants,  n  =  16  were  14E  (Patriot  Fire  Control  Enhanced 
Operator/Maintainer)  and  n  =  4  were  140E  (Warrant  Officer  MOS  Air  and  Missile  Defense 
Systems  Tactician/Technician).  Most  participants  were  Specialists  (E-4;  n  =  13)  who  were 
serving  in  TCA  positions  ( n  =  13).  The  participants  had  been  in  their  current  assignment  for  M  = 
14.6  months  (SD  =  16.5  months),  with  most  (55%)  in  their  current  assignment  for  six  or  fewer 
months.  Most  participants  (n  =  1 1)  were  certified  at  least  Level  VIII,  Intermediate- Level 
Gunnery  Certification,  which  is  the  requisite  certification  for  executing  a  real-world  mission  on 
the  Patriot  system. 


Results  of  the  Phase  III  Validation 
Background  Training  of  Participants 

Prior  to  the  experimental  training,  participants  completed  a  questionnaire  asking  about 
their  training  background  in  Patriot  (see  Appendix  C).  Participants  were  asked  to  rate  each 
statement  on  the  questionnaire  in  terms  of  frequency:  1  =  Never;  2  =  Occasionally;  3  =  Often; 
and  4  =  Always.  Table  7  presents  the  results. 

Table  7 


Participant  Responses  to  Pre-Training  Training  Background  Questionnaire 


Item 

Never 

Occasionally 

Often 

Always 

1. 

The  Patriot  engagement  training  received  in  my 

1 

4 

6 

9 

unit  has  been  focused  on  remembering  and 
understanding  procedures. 

(5%) 

(20%) 

(30%) 

(45%) 

2. 

My  Patriot  engagement  training  has  focused  on 

1 

2 

8 

9 

applying  the  procedures  that  I  can  recall  and 
understand. 

(5%) 

(10%) 

(40%) 

(45%) 

My  Patriot  training  has  provided  information  that  my  crew  and  I  were  required  to  analyze  and  evaluate 

information  and  data  in  order  to:  (See  3-14  below ) 

3. 

Set  work  and  task  priorities 

2 

4 

5 

9 

(10%) 

(20%) 

(25%) 

(45%) 

4. 

Review  and  modify  priorities 

5 

- 

6 

9 

(25%) 

(30%) 

(45%) 

5. 

Set  engagement  priorities  for  tracks  or  targets  on 

2 

3 

5 

10 

my  scope 

(10%) 

(15%) 

(25%) 

(50%) 

ANTICIPATE: 

6. 

Track  separations 

2 

4 

5 

9 

(10%) 

(20%) 

(25%) 

(45%) 

7. 

Launch  of  ASMs  or  ARAMs 

3 

1 

5 

10 

(1  missing) 

(15.8%) 

(5.3%) 

(26.3%) 

(52.6%) 

8. 

Jamming 

4 

2 

7 

7 

(20%) 

(10%) 

(35%) 

(35%) 

9. 

The  intent  of  a  track  or  a  hostile  formation 

3 

1 

9 

7 

(15%) 

(5%) 

(45%) 

(35%) 

10. 

Launcher  reload  requirements 

3 

2 

7 

8 

(15%) 

(10%) 

(35%) 

(40%) 
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Table  7 


Participant  Responses  to  Pre-Training  Training  Background  Questionnaire  ( Continued ) 


Item 

Never 

Occasionally 

Often 

Always 

11. 

Need  to  alter  system  settings  on  the  Patriot 

2 

3 

5 

10 

(10%) 

(15%) 

(25%) 

(50%) 

12. 

The  probable  sequence  of  events  during  a  hostile 

3 

3 

4 

10 

attack 

(15%) 

(15%) 

(20%) 

(50%) 

ACT  TO  COUNTER: 

13. 

Erroneous  classification  of  tracks 

3 

3 

8 

6 

(15%) 

(15%) 

(40%) 

(30%) 

14. 

Coordinated  SEAD  directed  against  my  battery 

5 

3 

5 

6 

(25%) 

(15%) 

(25%) 

(30%) 

Note:  N  =  20.  ASM  is  an  Air-to-Surface  Missile.  ARAM  is  an  Anti-Radiation  Missile.  SEAD  is  Suppression  of 
Enemy  Defenses. 


More  than  half  of  the  participants  reported  that  they  always  received  training  that 
prepared  them  to  analyze  and  evaluate  information  to  set  engagement  priorities  for  tracks  or 
targets  on  scope,  anticipate  the  launch  of  ASMs  or  ARAMs,  anticipate  the  need  to  alter  system 
settings  on  the  Patriot,  and  anticipate  the  probable  sequence  of  events  during  a  hostile  attack. 
The  fewest  participants  indicated  that  they  always  received  training  on  analyzing  and  evaluating 
information  to  act  to  counter  erroneous  classification  of  tracks,  and  coordinated  suppression  of 
enemy  defenses  (SEAD)  directed  against  their  battery. 

We  asked  additional  questions  concerning  cognitive  skills  training  the  participants  might 
have  received.  Sixteen  (80%)  participants  reported  they  had  been  provided  training  on  models 
and  methods  for  problem-solving  and  engagement  problems.  Thirteen  (65%)  reported  that  they 
had  been  provided  scenarios  and  situations  to  assess  their  critical  thinking  and  problem  solving, 
and  had  been  given  feedback.  Fifteen  (75%)  reported  that  they  had  been  trained  on  situations  to 
practice  critical  thinking  and  problem  solving  skills. 

User  Performance  within  Modules 

Following  the  pre-training  data  collection,  the  participants  then  completed  the  five 
training  modules.  On  completing  each  section,  participants  documented  their  progress  on  an  in¬ 
route  assessment  form.  The  details  of  the  in-route  assessment  are  presented  in  Appendix  D. 
Table  8  summarizes  the  participants’  performance  in  terms  of  raw  mean  and  percentage  scores 
for  each  of  the  five  modules. 
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Table  8 


Participant  Performance  for  Each  Training  Module 


Module/Problem  Set 

N 

M 

SD 

Low 

Score 

High 

Score 

Number 
of  Items 

Crawl  (Beginner) 

Problem  1 — Decision  for  TBM  track 

18 

3.44 

0.86 

2 

5 

5 

Problem  2 — Decision  for  Unknown  track 

18 

3.94 

1.06 

2 

5 

5 

Problem  3 — Decision  for  3  Unindentified  aircraft 

18 

4.83 

1.20 

2 

6 

6 

Total  Raw  Score 

18 

12.22 

2.37 

6 

16 

16 

Percent  Correct 

18 

76% 

15% 

38% 

100% 

- 

Walk  (Intermediate) 

Problem  1 — Multiple  aircraft  attacking 

18 

3.05 

1.47 

1 

6 

6 

Problem  2 — Determine  probable  enemy  actions 

18 

6.50 

2.66 

3 

12 

12 

Problem  3 — SCUDs  and  an  air  launch 

18 

5.72 

1.67 

3 

8 

8 

Total  Raw  Score 

18 

15.28 

4.44 

7 

24 

24 

Percent  Correct 

18 

64% 

19% 

29% 

100% 

- 

Run  Scenario  I  (Advanced) 

Problem  1 — Multiple  aircraft  attacking 

18 

2.94 

1.16 

1 

5 

7 

Problem  2 — Multiple  aircraft  attacking 

18 

4.33 

1.85 

1 

6 

6 

Problem  3 — Maintenance  and  reload 

18 

2.28 

1.02 

0 

4 

5 

Total  Raw  Score 

18 

9.95 

2.75 

4 

14 

18 

Percent  Correct 

18 

53% 

15% 

22% 

78% 

- 

Run  Scenario  II  (Advanced) 

Problem  1 — Current  threats  and  probable  actions 

18 

3.67 

2.25 

0 

8 

10 

Problem  2 — Multiple  TBMs  on  scope 

18 

2.00 

1.14 

0 

4 

4 

Problem  3 — Quickly  recover  mission  capabilities 

18 

2.33 

1.41 

0 

4 

5 

Total  Raw  Score 

18 

8.00 

3.82 

0 

15 

19 

Percent  Correct 

18 

42% 

20% 

0% 

79% 

- 

Run  Scenario  III  (Advanced) 

Problem  1 — Multiple  aircraft  attacking 

19 

2.47 

1.50 

0 

5 

7 

Problem  2 — Multiple  aircraft  attacking 

19 

2.68 

1.34 

0 

5 

6 

Problem  3 — Multiple  inbound  TBMs 

19 

2.16 

1.80 

0 

5 

6 

Total  Raw  Score 

19 

7.32 

3.45 

0 

14 

15 

Percent  Correct 

19 

48% 

23% 

0 

93% 

— 

Note  that  as  the  difficulty  of  scenarios  increased — from  crawl,  to  walk,  to  run — the 
participants’  percentage  scores  decreased  from  76%  in  the  crawl  phase  to  48%  in  the  run  phase. 
In  the  crawl  phase,  participants  made  simple  decisions,  identified  targets,  and  provided  a 
rationale  for  their  decisions.  In  the  walk  phase,  participants  prioritized  threats  in  multipronged 
attacks,  predicted  enemy  actions,  identified  targets,  and  provided  a  rationale  for  their  responses. 
In  the  run  phase,  participants  prioritized  threats  in  multipronged  attacks,  predicted  enemy 
actions,  identified  targets,  made  fix  or  fight  decisions,  and  provided  a  rationale  for  their 
responses.  We  manipulated  training  complexity  by  changing  the  number  of  elements 
participants  had  to  consider  in  their  decision-making  process,  as  well  as  the  types  of  decisions 
they  were  being  asked  to  make  (e.g.,  simple  identification  of  a  threat  vs.  prioritization  of  multiple 
threats). 
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User  Evaluation  of  the  Training  Supports 

As  each  phase  of  training  concluded,  participants  rated  how  well  scaffolding  features  of 
the  training  had  supported  them  in  addressing  the  presented  problems.  Table  9  summarizes  their 
responses. 

Table  9 


User  Evaluation  of  Training  Supports 


Training 

Module 

Content  Area 

Not 

Useful 

Somewhat 

Useful 

Useful 

Very 

Useful 

Essential 

Crawl* 

AAR/Feedback 

0 

5 

4 

6 

2 

(0%) 

(29%) 

(24%) 

(35%) 

(12%) 

Walk 

AAR/Feedback 

1 

3 

6 

5 

2 

(6%) 

(18%) 

(35%) 

(29%) 

(12%) 

Graphic  Organizer/Mental  Model 

1 

3 

4 

5 

4 

(6%) 

(18%) 

(24%) 

(29%) 

(24%) 

Additional  Resources 

1 

3 

6 

3 

4 

(6%) 

(18%) 

(35%) 

(29%) 

(24%) 

Run 

AAR/Feedback 

0 

4 

6 

4 

3 

(0%) 

(24%) 

(35%) 

(24%) 

(18%) 

Graphic  Organizer/Mental  Model 

0 

1 

4 

9 

3 

(0%) 

(6%) 

(24%) 

(53%) 

(18%) 

Additional  Resources 

0 

3 

4 

8 

2 

(0%) 

(18%) 

(24%) 

(47%) 

(12%) 

Introductory 

Training 

Task  and  Purpose 

1 

0 

6 

7 

3 

(6%) 

(0%) 

(35%) 

(41%) 

(18%) 

Graphic  Organizer/Mental  Model 

1 

1 

6 

6 

3 

(6%) 

(6%) 

(35%) 

(35%) 

(18%) 

Scenario/Problem  Presentation 

0 

1 

6 

5 

5 

(0%) 

(6%) 

(35%) 

(29%) 

(29%) 

General  Situation/ Additional 

0 

1 

6 

8 

2 

Resources 

(0%) 

(6%) 

(35%) 

(47%) 

(12%) 

*Note:  N  =  17;  at  the  conclusion  of  the  Crawl  phase  of  training.  Soldiers  were  asked  to  rate  the  Graphic 
Organizer/Mental  Model  and  Additional  Resources  in  terms  of  whether  or  not  they  assisted  with  their  decision¬ 
making  process.  Fifteen  out  of  17  Soldiers  (88%)  reported  that  the  graphic  organizer  assisted  their  decision-making 
process;  fourteen  out  of  17  Soldiers  (82%)  reported  that  the  additional  resources  assisted  their  decision-making 
process. 

Most  participants  found  the  scaffolding  features  useful  in  assisting  their  decision-making 
process.  Note  the  large  number  of  participants  who  found  the  graphic  organizer  and  additional 
resources  ‘very  useful’ — 53%  and  47%,  respectively — in  the  mn  phase  of  training.  In  the  run 
phase,  scaffolding  feedback  was  reduced  and  participants  had  to  make  multiple  decisions  before 
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receiving  an  AAR/feedback.  With  reduced  AAR/feedback,  participants  had  to  infer  that  they 
had  made  the  wrong  decision  based  on  the  consequences  of  their  decision  in  the  evolving 
scenario.  They  may  have  relied  more  on  the  graphic  organizer  to  support  the  evaluation  process 
in  the  absence  of  direct  AAR/feedback. 

User  Ratings  of  Learning  Outcomes 

After  training,  we  asked  participants  to  complete  a  questionnaire  concerning  various 
aspects  of  the  training  scenarios  and  their  learning  outcomes  (see  Appendix  E).  Most 
participants  (95%;  n  =  19  out  of  20)  reported  that  (a)  the  cognitive  skills  training  provided 
models  or  methods  for  problem  solving  and  engagement  problems,  (b)  that  the  training  provided 
scenarios  and  situations  to  assess  their  critical  thinking  and  problem  solving,  and  (c)  that  the 
training  provided  a  means  to  practice  their  critical  thinking  and  problem  solving  skills.  The 
participants  rated  the  scenarios  as  (a)  realistic  (90%;  n  =  18  out  of  20),  (b)  challenging  (75%;  n  = 
15  out  of  20),  (c)  frustrating  (40%;  n  =  8  out  of  20),  (d)  too  difficult  (10%;  n  =  2  out  of  20),  and 
(e)  too  easy  (5%;  n  =  1  out  of  20). 

Participants  were  asked  to  rate  their  understanding  and  skills  before  and  after  training  for 
a  set  of  relevant  topic  areas.  These  ratings  were  made  on  the  scale:  0  =  ‘No  Experience’;  1  -  3  = 
‘Can  Get  By’;  4  -  6  =  ‘Pretty  Good’;  and  7  =  ‘Can  Train  Others’.  The  differences  between 
participants’  before  and  after  ratings  were  examined  using  a  within-subjects  repeated  measures 
analysis  of  variance  (ANOVA).  The  partial  eta-squared  effect  size  measure  was  used.  Table  10 
presents  the  results  of  the  participants’  ratings. 
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Table  10 


Results  of  Soldier  Ratings  of  their  Understanding  and  Skills  Before  and  After  Training 


Item 

Rating 

Before  After 

M  (SD)  M  ( SD ) 

N 

F(df) 

P 

Hr2 

To  What  Degree  Did  the  Patriot  Cognitive  Skills 

Trainer  improve  on  your  understanding  of  the 

following: 

Setting  work  and  task  priorities 

3.94 

4.89 

18 

8.82 

0.009 

0.34 

(2.21) 

(1.64) 

0,  17) 

Reviewing  and  modifying  priorities 

3.94 

4.67 

18 

4.69 

0.019 

0.29 

(2.23) 

(1.81) 

0,  17) 

Setting  engagement  priorities  for  tracks  or  targets  on 

4.28 

5.06 

18 

5.95 

0.026 

0.26 

my  scope 

(2.52) 

(1.92) 

0,  17) 

Self-assess  your  understanding  and  skills  when 

reviewing  Patriot  system  data  and  anticipating: 

Track  separations 

4.72 

5.22 

18 

5.28 

0.035 

0.24 

(2.56) 

(1.90) 

0,  17) 

Launch  ofASMs  or  ARAMs 

4.78 

5.33 

18 

5.12 

0.037 

0.23 

(2.51) 

(1.78) 

0,  17) 

Jamming 

4.56 

5.06 

18 

2.89 

0.11 

0.15 

(2.41) 

(1.98) 

0,  17) 

The  intent  of  a  track  or  a  formation  of  hostile  aircraft 

4.44 

5.22 

18 

7.37 

0.02 

0.30 

(2.33) 

(1.77) 

0,  17) 

The  need  to  alter  system  settings  on  the  Patriot 

4.00 

4.56 

18 

3.86 

0.07 

0.19 

(2.42) 

(2.28) 

0,  17) 

The  probable  sequence  of  events  during  a  hostile 

4.28 

4.83 

18 

4.21 

0.06 

0.20 

attack 

(2.44) 

(2.01) 

0,  17) 

Self-assess  your  understanding  and  skills  when 

reviewing  Patriot  system  data  and  acting  to 

counter: 

Erroneous  classification  of  tracks 

3.72 

4.33 

18 

4.35 

0.05 

0.20 

(2.54) 

(2.43) 

0,  17) 

Coordinated  SEAD  directed  against  my  Patriot 

3.00 

3.50 

18 

3.40 

0.08 

0.17 

system  and  my  battery 

(2.67) 

(2.60) 

0,  17) 

Soldiers  indicated  that  they  perceived  improvement  in  their  understanding  and  skills  in  six  (6)  of 
the  1 1  areas  evaluated.  Figure  8  presents  results  for  these  areas  of  perceived  improvement. 
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Figure  8.  Six  Areas  of  Perceived  Improvement  Attributed  to  the  Patriot  Cognitive  Skills  Trainer. 


Based  on  the  results  of  the  within-subjects  repeated  measures  ANOVA,  participants 
perceived  significant  improvement  in  their  understanding  and  skills  regarding  (a)  setting  work 
and  task  priorities,  (b)  setting  engagement  priorities  for  tracks  or  targets  on  scope,  (c) 
anticipating  the  intent  of  a  track  or  formation  of  hostile  aircraft,  (d)  reviewing  or  modifying 
priorities,  (e)  anticipating  launch  of  ASMs  or  ARAMs,  and  (f)  anticipating  track  separations. 
These  results  appear  to  relate  to  two  key  cognitive  skills:  anticipating  events  and  prioritizing 
actions. 


User  Evaluation  of  the  Patriot  Cognitive  Skills  Trainer 

Concluding  the  data  collection,  participants  were  asked  to  rate  the  training  content  and 
quality  of  their  learning  experience.  The  Training  Module  Rating  Questionnaire  was  adapted 
from  a  six-factor  computer-based  training  evaluation  developed  by  Graves,  Blankenbeckler, 
Wampler,  and  Roberts  (2016).  The  ratings  covered  six  topic  areas:  (a)  Quality  of  Learning 
Experience,  (b)  Quality  of  Design  and  Content,  (c)  Continuity  of  Topics,  (d)  Credible  Examples, 
(e)  Focus  and  Relevance,  and  (f)  User  Ability  to  Track  Progress.  Participants’  detailed  responses 
are  presented  in  Appendix  F.  Participants  were  asked  to  rate  statements  concerning  each  of  the 
six  topic  areas  on  a  standard  5-point  Likert  scale  from  ‘Strongly  Disagree’  =  1  to  ‘Strongly 
Agree’  =  5.  Soldiers’  responses  on  the  Training  Module  Rating  Questionnaire  are  summarized 
in  Table  11. 
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Table  11 


Soldiers’  Responses  on  the  Training  Module  Rating  Questionnaire 


Factor 

N 

M 

SD 

Low 

High 

Quality  of  Learning  Experience 

20 

3.66 

0.86 

1.86 

5.00 

Quality  of  Design  and  Content 

20 

4.02 

0.59 

2.86 

5.00 

Continuity  of  Topics 

20 

4.13 

0.57 

2.80 

5.00 

Credibility  of  Examples 

20 

3.86 

0.86 

2.00 

5.00 

Focus  and  Relevance  of  Training 

20 

4.07 

0.66 

2.75 

5.00 

Ability  to  Track  Progress  in  Training 

20 

4.45 

0.51 

4.00 

5.00 

Participants’  ratings  of  the  training  tended  to  be  positive.  The  highest  rated  areas 
concerned  (a)  the  participants’  ability  to  track  their  own  progress  when  executing  the  training, 

(b)  the  continuity  of  topics  presented  in  the  training,  (c)  the  focus  and  relevance  of  the  training, 
and  (d)  the  quality  of  training  design  and  content.  Although  also  in  the  positive  direction,  the 
lowest  rated  areas  were  factors  related  to  (a)  the  quality  of  the  Soldiers’  learning  experience  and 
(b)  the  credibility  of  examples.  Based  on  Soldiers’  comments,  some  found  the  training  difficult 
in  that  there  was  no  solution  that  was  absolutely  “correct”  for  the  types  of  problems  the  training 
presented.  Some  Soldiers  found  themselves  arguing  with  the  decision  and  the  rationale  of  the 
SMEs  against  which  their  own  decisions  and  rationale  were  being  scored.  For  two  participants, 
this  created  observable  frustration — particularly  when  Soldiers  had  deprioritized  a  novel  threat  in 
the  Run  phase  of  the  training  that  the  SMEs  had  perceived  to  be  far  more  salient  (i.e.,  swarming 
HARPYs).  In  different  training  situations,  this  discrepancy  could  be  turned  into  an  informative 
discussion  about  decision-making  and  problem  solving  in  the  Patriot  operational  context. 

User  Comments 

Soldiers  were  asked  to  provide  feedback  on  how  the  PCST  could  be  improved.  The 
following  comments  were  of  particular  interest: 


•  “Some  answers  were  dissimilar  to  SMEs  due  to  battery  SOPs.”  Trainers/instructors 
could  address  this  issue  in  AARs  and  discussions  following  the  training.  When  the  SMEs 
were  presented  with  the  problem  sets  covered  in  the  training,  they  were  told  to  answer 
from  the  perspective  of  doctrine  and  the  ROE  presented  in  the  training.  While  unit  SOP 
and  ROE  will  differ  between  units  and  particular  engagements,  we  sought  to  set  the 
context  of  the  scenarios  as  generally  as  possible  to  broaden  the  applicability  of  the 
training  to  as  many  units  as  possible. 

•  “Maybe  have  a  more  experienced  TCA  paired  with  a  less  experienced  TCA  to  clarify 
unclear  things.”  This  is  an  excellent  suggestion  for  how  to  utilize  the  PCST  in  a  battery 
training  context.  In  one  case,  two  Warrant  Officers  were  working  through  the  training 
together.  This  led  to  a  number  of  informative  discussions  not  only  about  the  problem  at 
hand,  but  the  rationale  presented  for  the  decisions  and  rationale  of  the  SMEs. 

•  “Make  sure  all  the  info  is  up-to-date  and  the  narrative  is  up-to-date”;  “Familiarity  with 
some  threats  and  aircraft”;  “Up-to-date  kinematics.”  Because  the  training  needed  to  be 
unclassified,  we  used  somewhat  dated  kinematics  and  hardware  for  the  opposing  force — 
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using  information  available  in  open  sources.  The  specified  technological  capabilities  of 
the  opposing  force  were  intended  to  mirror  what  Soldiers  may  encounter  in  a  real  world, 
operational  context. 


Discussion 

This  research  targeted  the  need  to  provide  ADA  Soldiers  with  cognitive  skills  training 
related  to  fire  control  decision  making  in  the  Patriot  system.  The  training  was  focused  on  new 
TCAs  and  TCOs,  serving  as  Patriot  crewmembers.  Our  research  resulted  in  the  POST,  a 
computer-based  training  package  that  was  intended  to  familiarize  less  experienced  air  defenders 
with  the  types  of  complex  problem  solving  and  reasoning  skills  that  expert  air  defenders  bring  to 
bear  in  making  their  engagement  decisions. 

Following  the  fratricide  incidents  in  OIF  I,  it  was  noted  that  crews  required  training  to 
develop  their  situational  awareness  and  vigilance  with  respect  to  launch  control  decision  making. 
The  Patriot  system  is  highly  automated  and  complex;  crews  require  the  expertise  to  understand 
the  significance  of  the  information  the  system  is  presenting  to  them  in  order  to  make  effective 
decisions  (Hawley,  2011).  This  requires  cognitive  skills  related  to  analyzing  and  evaluating 
information,  creating  solutions,  and  anticipating  outcomes. 

The  research  and  training  development  problem  was  addressed  in  three  phases.  In  the 
first  phase,  we  gathered  information  on  the  types  of  problems  that  Patriot  crews  encounter  and 
find  difficult  to  resolve,  and  sought  to  develop  a  training  solution  to  address  their  identified 
needs.  In  the  second  phase  of  the  research,  we  developed  the  training  based  on  our  findings  from 
the  first  phase  of  the  research.  In  the  third,  and  final,  phase  of  the  research,  we  validated  the 
PCST  with  Soldiers  at  the  108th  ADA  Brigade  at  Fort  Bragg,  NC. 

Considerations  for  Training  with  the  Patriot  Cognitive  Skills  Trainer 

The  PCST  is  a  computer-based  training  program  intended  to  assist  new  Patriot  TCOs  and 
TCAs  in  learning  the  complex  decision-making  and  problem  solving  skills  of  more  experienced 
Patriot  operators.  We  envisioned  that  the  trainer  would  be  used  in  Schoolhouse  and  unit  training 
settings,  incorporating  it  into  existing  training  to  help  structure  a  broader  approach  to  developing 
decision-making  and  problem  solving  skills.  One  advantage  to  the  unclassified  nature  of  the 
training  is  that  a  unit  trainer  or  instructor  can  provide  the  training  on  a  disk  for  a  learner  to 
execute  on  his  or  her  own.  The  feedback  from  each  section  can  be  printed  and  used  to  facilitate 
later  discussions  between  the  unit  trainer/instructor  and  the  learner. 

The  types  of  problems  the  learner  encounters  within  the  PCST  are  rare.  To  encounter 
similar  decision-making  situations  and  problems  in  normal  operational  experience  could  take 
many  years.  The  PCST  was  designed  to  familiarize  Patriot  operators  earlier  in  their  careers  with 
the  types  of  unfamiliar  decision-making  situations  and  problems  they  may  encounter  in  order  to 
give  them  insight  into  how  more  experienced  operators  think  about  and  work  through  similar 
issues.  There  are  no  absolutely  correct  or  incorrect  answers  to  these  types  of  ill-defined 
problems;  what  the  learner  is  being  trained  to  do  is  how  to  think  about  these  types  of  problems. 

A  similar  training  outcome  is  targeted  in  SPEAR.  While  the  PCST  does  not  have  the  on-the-fly 
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adaptability  of  SPEAR  training,  it  could  be  used  as  part  of  the  preparation  for  the  novel  problem 
solving  situations  Soldiers  may  experience  when  engaged  in  a  SPEAR  training/certification. 

The  purpose  of  the  PCST  is  to  provide  a  supplement  to  traditional  Patriot  training,  which 
tends  to  focus  on  procedural  skills  to  achieve  tactical  and  technical  mastery  of  the  Patriot  system. 
The  users  of  the  PCST  need  to  have  some  understanding  of  the  context  of  Patriot  operations  and 
know  how  to  read  and  interpret  an  air  battle  on  scope.  They  do  not  require  specific  knowledge 
of  how  to  execute  technical  procedures  with  the  Patriot  system.  In  this  way,  the  trainer  targets 
the  cognitive  skills  needed  to  make  sense  of  an  air  battle,  interpret  changes  in  information,  and 
make  decisions  based  on  that  information.  The  additional  cognitive  load  of  operating  the 
technology  is  reduced  in  the  trainer,  allowing  learners  to  focus  on  critical  cognitive  skills 
development. 

In  developing  the  PCST  we  did  not  seek  to  simulate  sensitive  capabilities,  processing 
capabilities,  or  embedded  resources  of  the  operational  Patriot  system.  This  choice  was  made  for 
two  reasons.  First,  many  of  the  technical  capabilities  and  procedures  for  operating  the  system 
are  classified.  Second,  we  wanted  Soldiers  to  focus  on  the  air  battle  and  events  shaping  the 
context  of  the  air  battle.  The  information  used  in  developing  the  scenarios  and  specifying  the 
capabilities  of  enemy  threats  was  drawn  from  open  source  literature.  It  was  determined  that 
having  the  most  up-to-date,  and  classified,  information  to  enhance  the  realism  of  the  training 
would  ultimately  yield  little  additional  benefit  to  developing  Soldiers’  cognitive  skills 
development.  The  requirement  for  developing  decision-making  and  problem  solving  skills 
would  necessitate  a  plausible  set  of  scenarios  and  problems  and  a  believable  context,  but  not 
faithful  emulation  of  the  Patriot  system  and  the  particulars  of  current  ROE  in  real-world 
deployment  locations. 

Further  supporting  this  focus  on  cognitive  skills  development  is  the  application  of  the 
METT-T  framework  to  assist  crewmembers  in  identifying  relevant  information  during  their 
decision-making  process.  This  framework  was  intended  to  facilitate  the  learner  in  visualizing 
their  immediate  problem  and  its  context.  Through  repetition  of  the  METT-T  framework  across 
the  various  problem  sets,  and  reduction  in  scaffolding  feedback,  the  learner  is  encouraged  to  rely 
more  and  more  on  the  model  to  organize  information  and  think  through  the  relevant  aspects  of 
their  present  problem. 

Finally,  the  training  was  designed  to  escalate  in  complexity  and  difficulty  as  the  learner 
progresses.  This  was  accomplished  by  introducing  additional  decision-triggering  events  along 
with  reduced  information,  scaffolding,  and  feedback.  Although  our  validation  of  the  training 
took  place  in  a  single  session,  a  trainer/instructor  would  likely  be  able  to  yield  better  results  by 
spreading  the  training  across  multiple  sessions.  This  would  provide  trainers/instructors  and 
learners  with  time  to  engage  in  more  in-depth  discussions.  These  discussions  would  support 
learners  in  learning  to  evaluate  scenarios  of  increasing  complexity  as  well  as  to  evaluate  and 
discuss  their  own  decisions  and  rationales  as  well  as  those  of  the  SMEs  presented  in  the  training. 
This  would  allow  a  trainer/instructor  to  address  specific  topics  concerning  how  unit  SOP  and 
ROE  may  differ  from  what  is  presented  in  the  training,  and  how  those  differences  may  impact 
decision-making  and  problem  solving  in  a  concrete  operational  context. 
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In  the  second  phase  of  the  project,  the  researchers  identified  context  specific  decision¬ 
triggering  events  around  which  to  design  the  problem  sets  presented  in  the  trainer.  Identifying 
context  specific  decision-triggering  events  seemed  to  be  an  effective  technique  for  identifying 
the  types  of  problems  that  would  have  both  face  validity  to  learners  and  elicit  the  complex 
cognition  necessary  for  training  decision-making  and  problem  solving  skills  for  the  Patriot 
context.  A  similar  technique  would  be  applicable  across  Army  training  domains  to  elicit 
decision-triggering  events  around  which  cognitive  skills  training  could  be  developed.  What  is 
key  in  the  process  is  identifying  problems  that  elicit  a  cognitive  rather  than  procedural  response. 
Establishing  a  viable  setting  and  context  for  a  problem  is  critical  in  this  respect.  For  instance,  in 
Patriot  operations  a  fix  or  fight  decision  will  only  arise  when  there  are  both  enemy  targets  on 
scope  and  a  degraded  technological  capability  in  the  Patriot  system.  Under  those  circumstances, 
operators  are  forced  to  decide  between  addressing  the  degraded  capability,  or  whether  they  can 
still  effectively  engage  the  immediate  threat  on  scope  first  and  then  address  the  degraded 
capability  at  another  time.  Working  with  a  set  of  domain  specific  decision-triggering  events,  a 
training  designer  can  derive  a  number  of  different  types  of  problem  scenarios  that  will  be  novel 
and  plausible  to  knowledgeable  learners  and  will  present  a  sufficient  challenge  to  their  decision¬ 
making  and  problem  solving  skills. 

Validation  of  the  Patriot  Cognitive  Skills  Trainer 

Overall,  the  group  of  Soldiers  we  tested  the  trainer  with  at  the  108th  ADA  BDE 
responded  positively  to  the  training.  While  the  sample  was  small  ( N  =  20),  it  was  an  ideal  mix 
of  TCAs  with  varying  levels  of  experience  and  certification,  as  well  as  some  warrant  officers 
with  TCA/TCO  experience.  The  full  data  collection  session  required  approximately  two-and-a- 
half  to  three  hours.  This  would  not  be  an  ideal  period  for  conducting  the  training  in  a  regular 
unit.  As  noted  above,  it  would  be  better  to  spread  the  training  across  multiple  sessions  to  allow 
for  discussion  between  trainers/instructors  and  learners  to  evolve. 

Given  our  findings  in  the  first  phase  of  the  research  that  most  Patriot  training  is  focused 
on  tactical  and  technical  procedures  specific  to  the  system,  it  was  notable  that  many  Soldiers  in 
the  validation  reported  ‘often’  or  ‘always’  receiving  training  related  to  analyzing  and  evaluating 
information  to  perform  various  Patriot  tasks.  Consistently,  more  than  50%  Soldiers  from  the 
108th  ADA  BDE  reported  receiving  training  related  to  analyzing  and  evaluating  information  as 
well  as  anticipating  and  acting  to  counter  events  such  as  track  separations,  erroneous 
classification  of  tracks,  launch  of  ASMs  or  ARAMs,  or  the  intent  of  a  hostile  formation.  Eighty 
percent  (80%)  of  participating  Soldiers  indicated  that  they  had  been  provided  training  on  models 
and  methods  for  problem  solving  and  engagement  problems;  65%  reported  being  provided 
scenarios  and  situations  to  assess  their  critical  thinking  and  problem  solving  and  had  been  given 
feedback;  and  75%  reported  being  trained  on  situations  to  practice  critical  thinking  and  problem 
solving  skills.  Given  these  findings,  one  may  conclude  that  the  sample  of  Soldiers  who 
participated  in  the  evaluation  of  the  PCST  were  not  naive  to  training  focused  on  decision-making 
and  problem  solving.  Without  directly  observing  the  in-unit  training,  however,  our  conclusions 
about  the  degree  to  which  higher-order  thinking  skills  were  actually  addressed  are  limited.  This 
set  of  results  may  reflect  a  command  emphasis  on  training  these  skills  at  the  unit  level,  but  how 
the  reported  in-unit  training  was  executed  is  unknown. 
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Even  given  the  participants’  potential  familiarity  with  training  for  decision-making  and 
problem  solving  in  the  Patriot  context,  the  participating  Soldiers  appeared  to  be  sufficiently 
challenged  by  the  scenarios  and  problem  sets  presented  by  the  PCST.  Most  were  very  successful 
in  the  crawl  phase  of  the  training,  averaging  scores  of  76%,  including  some  perfect  scores  of 
100%.  In  the  walk  phase  of  training,  when  they  were  presented  with  more  complex  problems, 
the  participants’  average  scores  dropped  to  64%,  although  some  still  achieved  perfect  scores  of 
100%.  Participants  appeared  to  have  the  greatest  difficulty  with  the  run  phase  of  training,  which 
presented  them  with  even  more  complex  problems,  reduced  information,  and  reduced 
scaffolding/feedback.  In  the  first  run  scenario,  participants  averaged  53%;  dropping  to  42%  in 
the  second  run  scenario;  and  rebounding  slightly  to  48%  in  the  third  run  scenario.  This  pattern  of 
results  led  us  to  conclude  that  the  training  may  be  effective  in  challenging  Soldiers  who  report 
receiving  some  background  training  on  decision-making  and  problem  solving  skills  specific  to 
the  Patriot  operational  context.  The  PCST  may  be  useful  to  challenge  Soldiers  in  units  even 
where  there  already  exists  some  emphasis  on  training  these  skills.  Soldiers  may  benefit  from  in- 
depth  discussions  with  peers  and  instructors  following  the  very  challenging  run  phase  scenarios. 

To  derive  additional  information  about  participants  reaction  to  the  training,  we  asked 
them  to  rate  how  they  perceived  their  skills  to  have  changed  from  before  the  training  to  after  the 
training.  While  this  is  not  an  ideal  approach  to  looking  at  the  longer-term  skill  development 
attributable  to  the  training,  it  was  a  more  feasible  approach  compared  to  alternatives  such  as  a 
study  of  far  transfer,  requiring  tracking  of  participants  and  later  additional  data  collections.  In 
six  of  the  1 1  areas  evaluated,  participants  reported  that  they  perceived  improvement  in  their 
understanding  and  skills  after  training.  These  six  areas  mainly  focused  on  anticipating  enemy 
actions  and  establishing  and  revising  priorities.  It  is  notable  that  they  did  not  indicate  perceived 
improvement  in  two  areas  that  were  directly  related  to  problem  sets  within  the  training — a 
problem  that  focused  on  overcoming  enemy  jamming  of  the  Patriot  radar,  and  another  problem 
related  to  a  direct  SEAD  against  their  battery.  Both  of  these  problems  were  introduced  in  the  run 
phase  of  training  and  may  indicate  useful  topics  for  additional  face-to-face  discussion  with 
trainers/instructors. 

Overall,  Soldiers  rated  the  training  positively,  particularly  in  terms  of  (a)  their  ability  to 
track  their  own  progress  when  executing  the  training,  (b)  the  continuity  of  topics  presented  in  the 
training,  (c)  the  focus  and  relevance  of  the  training,  and  (d)  the  quality  of  training  design  and 
content.  While  still  positive,  the  lowest  rated  areas  concerned  (a)  the  quality  of  the  Soldiers’ 
learning  experience  and  (b)  the  credibility  of  examples.  Participants’  comments  may  explain 
these  lower  ratings,  particularly  those  related  to  how  the  PCST  differed  from  unit  SOP  and  that 
the  technology  employed  in  the  scenarios  was  “out-of-date.”  These  ratings  reflect  the  trade-off 
we  made  in  order  to  keep  the  Trainer  in  the  unclassified  domain.  We  focused  the  training  on 
general  Army  doctrine — reducing  the  impact  of  SOP  and  theater- specific  ROE — and  used  open 
source  materials  to  gather  information  about  the  technical  capabilities  of  the  systems  being 
employed  in  the  scenarios. 


Conclusions 

This  research  demonstrates  it  is  possible  to  train  complex  cognitive  skills  for  Patriot  fire 
control  decision-making  and  problem  solving  using  hypothetical  scenarios  and  problem  sets,  and 
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unclassified  technical  information.  The  critical  aspect  of  developing  this  type  of  training 
concerns  developing  a  list  of  viable  decision-triggering  events,  that  are  both  challenging  and 
context  specific.  Training  scenarios  can  be  designed  to  introduce  decision-triggering  events  in 
various  sequences  and  rates  to  increase  the  complexity  of  the  training  that  Soldiers  are  working 
through.  In  addition,  feedback  and  scaffolding  support  can  be  augmented  or  diminished  to 
manipulate  the  challenge  presented  to  learners  in  the  training.  Finally,  even  when  training 
designers  are  faced  with  ambiguous  or  ill-defined  problems,  they  can  design  feedback  for 
learners  around  the  reactions  of  more  experienced  SMEs  to  the  same  set  of  problems.  This 
provides  a  point  of  comparison  for  less  experienced  learners  to  gain  insight  into  the  decision¬ 
making  and  problem  solving  processes  of  those  who  are  more  experienced  in  a  domain. 

Based  on  these  results,  we  expect  that  the  PCST  may  be  used  to  supplement  existing 
training  in  the  ADA  Schoolhouse  and  at  the  unit  level  to  support  the  development  of  skills 
related  to  decision-making  and  problem  solving  for  Patriot  operations.  While  the  PCST  may  be 
used  for  stand-alone  training,  the  most  benefit  will  likely  be  derived  when  it  is  incorporated  into 
a  training  situation  that  will  allow  for  additional  discussion  between  trainers/instructors  and 
learners. 

Key  Limitations 

A  primary  limitation  of  this  research  was  the  small  sample  size  used  in  the  validation.  A 
preferable  approach  would  have  included  a  larger  number  of  Soldiers  from  a  variety  of  units. 
With  a  broader  sample,  variations  in  organization-specific  training  emphases  and  practices  would 
have  been  mitigated.  Moreover,  a  longer  term  of  evaluation  would  have  enabled  us  to  evaluate 
training  transfer.  We  can  infer  from  the  results  of  our  field  test  with  the  PCST  that  Soldiers  felt 
they  learned  from  the  training,  but  we  cannot  definitively  say  that  their  decision-making  and 
problem  solving  performance  in  operational  contexts  has  been  improved.  In  order  to  determine 
this,  future  research  could  look  toward  executing  longitudinal  research  emphasizing  the  transfer 
of  PCST  training  to  the  operational  context. 
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A  Brief  Discussion  of  the  Air  Defense  Mission 


The  ADA  mission  is  “to  protect  the  force  and  selected  geopolitical  assets  from  aerial 
attack,  missile  attack,  and  surveillance”  (Department  of  the  Army,  2007).  The  ADA  mission 
makes  use  of  a  complex  set  of  lethal  missile  systems  and  sensors,  among  which  the  Patriot 
surface-to-air  missile  system  is  key  to  their  mission  (Department  of  the  Army,  2010a).  Patriot 
batteries  and  battalions  are  normally  task  organized  as  part  of  an  IAMD  TF  reinforcing  a 
supported  unit’s  AMD  capability  and  enhancing  force  protection.  These  TFs  may  also  include  a 
combination  of  Army  or  Joint  air  defense  systems.  A  TF  may  consist  of  a  combination  of 
command  and  control,  detection  and  tracking,  and  units  with  their  weapon  systems.  These  may 
include: 

•  Patriot, 

•  Terminal  High- Altitude  Area  Defense  (THAAD), 

•  Short  range  air  defense  systems, 

•  Counter  rockets,  artillery,  and  mortar  system, 

•  Long  range  alert  and  tracking  radar  systems, 

•  Area  air  defense  command  and  control  systems,  and 

•  Other  Joint  and  multinational  capabilities  or  units. 

The  composition  of  an  IAMD  task  force  is  based  on  specific  mission  requirements,  units,  and 
systems  available. 

While  conventional  thinking  may  view  IAMD  as  primarily  a  wartime  or  contingency 
operation,  today  U.S.  Army  Patriot  units  play  a  vital  role  in  deterrence  through  “Patriot 
Diplomacy.”  The  forward  deployed  presence  and  the  familiar  silhouettes  of  Patriot  launchers  on 
foreign  soil  provide  tangible  evidence  of  U.S.  commitment  to  allies,  a  resource  to  preserve  peace 
and  deter  hostilities  in  potentially  unstable  regions.  Army  Patriot  units  are  currently  deployed  in 
operational  missions  across  the  globe,  providing  protection  to  allied  cities,  transport  and  logistic 
facilities,  waterways,  and  air  corridors  as  well  as  forward  deployed  U.S.  forces  and  facilities. 

The  proximity  of  these  forward  deployed  units  to  potentially  hostile  nations  and  the  relative  short 
flight  times  of  available  strike  aircraft  and  missile  systems  of  these  nations,  keep  Patriot 
crewmembers  only  seconds  away  from  the  front  page  of  tomorrow’s  newspaper  or  a  flash 
bulletin  on  one  of  the  24-hour  news  channels  should  their  mission  of  deterrence  fail. 

For  contingency  operations  or  war,  Patriot  may  be  present  in  the  AMD  force  package  of  a 
mature  or  maturing  theater  of  operations  (Department  of  the  Army,  2008a).  For  example,  initial 
entry  of  U.S.  forces  into  an  opposed  or  unopposed  theater  is  generally  characterized  by  a  buildup 
through  the  introduction  of  tailored  force  packages  and  elements.  For  the  Patriot,  this  initial  air 
or  sea  deployable  element  may  be  as  small  as  a  minimal  engagement  package  (MEP)  (O’Neil, 
2011).  A  basic  MEP  consists  of  an  engagement  control  station  (ECS),  a  radar  set  (RS),  two  or 
more  launchers,  electric  power  plant,  repair  parts,  limited  utility  and  cargo  vehicles,  a  fuel 
tanker,  guided  missile  transporters,  and  a  mission  tailored  load  of  missiles.  A  MEP  deploys  with 
sufficient  supporting  equipment,  munitions,  supplies,  rations,  and  personnel  to  sustain  24-hour 
operations  for  only  a  limited  period.  The  exact  composition  is  mission  and  situation  dependent. 
However,  for  the  Soldiers  manning  a  MEP  the  responsibilities  for  protection  of  the  supported 
force  against  air  and  missile  attacks  are  enormous. 
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In  contrast,  a  full  Patriot  battalion  may  consist  of  a  headquarters  battery,  a  maintenance 
company,  and  between  four  and  six  firing  batteries  along  with  all  of  their  primary  weapons 
systems  and  supporting  equipment  (Department  of  the  Army,  2010a).  Current  configurations 
include  “pure”  battalions  and  mixed  battalions  providing  a  combination  of  Patriot,  the  short 
range  Avenger,  and  THAAD  missile  systems.  A  Patriot  firing  battery  generally  consists  of  six 
launching  stations,  RS,  ECS,  electric  power  plant,  and  one  antenna  mast  group  (AMG).  The 
Patriot  units’  ability  to  simultaneously  engage  large  numbers  of  attacking  aircraft,  theater 
ballistic  missile  (TBM)  systems,  standoff  jammers,  and  specific  aircraft  at  relatively  long  ranges, 
allows  the  supported  commander  increased  freedom  to  execute  the  close  as  well  as  the  deep 
battle  with  reliable  protection  against  enemy  aircraft  and  missile  attacks  (Department  of  the 
Army,  2008a). 

While  there  are  currently  several  system  and  missile  variants,  two  primary  missile  types 
provide  defensive  lethality  to  the  Patriot  system:  1)  the  Patriot  Advanced  Capability  (PAC)-2  and 
2)  PAC-3  missiles  (Department  of  the  Army,  2010a). 10  Varied  engagement  techniques  are 
available  to  the  ECS  crew. 

Patriot  may  be  employed  to  execute  a  variety  of  AMD  roles.  These  roles  include,  but  are 
not  limited  to: 

•  Executing  long  and  medium  range  engagements  against  manned  and  unmanned  enemy 
aircraft,  airborne  jammers,  and  reconnaissance  systems; 

•  Bounding  forward  with  attacking  or  maneuvering  forces  to  provide  protection  for 
elements  within  an  axis  of  advance  or  priority  sector; 

•  Protecting  defending  or  stationary  forces  and/or  facilities  with  overlapping  fires,  defense 
in  depth,  and  weighted  coverage; 

•  Protecting  AMD  radars  and  installations  against  attack  by  air-to-surface  (ASM),  anti¬ 
radiation  missiles,  and  land  attack  cruise  missile  (LACM)  systems;  and 

•  Providing  single  system  TBM  defense  or  the  lower-tier  of  integrated  TBM  defenses. 

Patriot  fire  units  (FUs)  acquire  and  track  the  incoming  threats  using  strategic  and  theater 
detection  and  interceptor  systems  and  organic  sensors  (Department  of  the  Army,  2010a).  AMD 
operations  are  characterized  by  centralized  control  and  planning  and  decentralized  execution  and 


10  The  PAC-2  configuration  is  the  larger  of  the  two  and  provides  a  single  missile  in  a  canister  with  a  load  of  four 
canisters  per  launcher.  PAC-2  is  a  single-stage,  solid-fuel,  ground-launched  interceptor  designed  to  destroy  aircraft, 
cruise  missiles,  or  tactical  ballistic  missiles  with  a  conventional  high-explosive  warhead  and  proximity  fusing.  With 
a  maximum  speed  of  over  3,500  mph  (1.56  kilometers  per  second),  it  operates  between  1.5  and  20  miles  (105,600 
feet/32  kilometers)  altitude  and  has  a  range  in  excess  of  60  miles  (96.5  kilometers).  While  optimized  for  missile 
engagements,  the  PAC-2  has  a  primary  role  against  high  performance  aircraft  and  other  threat  air  breathing  threats. 
The  PAC-3  was  designed  from  inception  as  an  agile,  high  velocity  missile  interceptor.  It  defeats  incoming  targets 
by  direct  body-to-body  impact.  However,  it  employs  a  lethality-enhanced  explosive  warhead  with  a  radial  pattern  of 
fragments  to  increase  missile  cross-section  and  enhance  kill  probability.  The  PAC-3  configuration  provides  four 
missiles  in  a  canister  with  four  canisters  (16  missiles)  per  Patriot  launch  station. 
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engagements.  Depending  on  the  directed  alert  state11  and  weapons  control  status12,  the  crew 
takes  the  handoff  of  hostile  tracks  as  they  come  within  range,  threaten  a  protected  asset,  or  when 
engagements  are  directed  against  specified  tracks  or  track  types.  Aircraft  engagements  are 
generally  performed  under  centralized  control  to  optimize  fires  as  well  as  to  minimize  fratricide 
risks.  Missile  engagements  may  be  executed  by  Patriot  units  alone  or  in  coordination  with 
THAAD  units,  with  Patriots  operating  as  the  lower  tier  of  a  two-tier  system.  In  these  integrated 
TBM  defenses,  coordination  is  required  to  optimize  the  use  of  interceptor  resources  as  well  as  to 
ensure  the  required  level  of  protection  (LOP)13  for  resources  and  units  being  defended. 

Engagements  by  the  Patriot  FU  are  controlled  and  executed  by  the  battery  ECS.  Figure  1 
provides  an  example  of  the  FU’s  deployment  and  the  battery  data  network.  Supported  by  varied 
radio  and  cable  communications  systems,  the  data  network  provides  connectivity  between  the 
ECS,  the  RS,  and  its  launchers,  as  well  as  the  ECS  and  the  broader  AMD  system.  This  data 
network  is  used  to  monitor  tracks,  launch  missiles,  establish  missile  availability,  and  monitor 
launch  station  status. 


11  Alert  states  represent  the  degree  of  readiness  of  ADA  units,  from  the  time  of  alert  notification,  to  the  time  of 
engagement  capability  or  battle  stations.  The  decision  as  to  which  to  degree  of  readiness  to  implement  is  situation 
dependent  and  determined  by  the  commander  in  coordination  with  higher  headquarters  or  the  controlling  authority, 
as  appropriate.  Alert  states  may  be  used  to  specify  personnel  and  manning  requirements.  Utilizing  alert  states 
allows  for  maximum  flexibility  to  conduct  training,  sustainment  operations,  or  perform  other  duties  while  meeting 
mission  requirements. 

12  Weapon  control  statuses  (Weapons  Free,  Weapons  Tight,  Weapons  Hold)  describe  the  relative  degree  of 
control  of  air  defense  fires.  Weapon  control  statuses  may  apply  to  weapon  systems,  volumes  of  airspace,  or  types  of 
hostile  air  platforms  or  missiles.  The  degree  or  extent  of  control  varies  depending  on  the  tactical  situation. 
Establishment  of  separate  weapon  control  statuses  for  fixed  and  rotary  wing  aircraft,  UAV,  and  missiles  is  normal. 
Air  and  missile  defense  forces  have  the  ability  to  receive  and  disseminate  weapon  control  statuses  for  all  classes  of 
air  platforms.  Weapons  Free  is  the  least  restrictive.  Weapons  can  fire  at  any  air  target  not  positively  identified  as 
friendly.  Weapons  Tight  limits  engagements  to  targets  that  are  identified  as  hostile  according  to  the  prevailing 
hostile  criteria.  Identification  can  be  effected  by  a  number  of  means  to  include  visual  identification  (aided  or 
unaided),  electronic,  or  procedural  means.  Capabilities  dictate  that  ADA  units  engage  TBMs  and  ASMs  based  on 
classification,  not  identification.  Weapons  Hold  is  the  most  restrictive  weapon  control  status  and  units  are  not  to 
engage  except  in  self-defense  or  in  response  to  a  formal  order. 

13  For  a  detailed  explanation  of  the  integrated  TBM  defense,  levels  of  TBM  operational  engagement  effectiveness, 
and  LOP  see  Chapter  3  of  ATTP  3-01.87  (Department  of  the  Army,  2010b) 
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Appendix  A,  Figure  1.  Patriot  Battery  Data  Net  (adapted  from  page  C-12  of  FM  3-01 .85) 

During  FU  operations,  the  ECS  is  the  only  manned  piece  of  equipment.  It  is  operated  by 
a  crew  of  three,  consisting  of  one  tactical  control  officer  (TCO),  one  tactical  control  assistant 
(TCA),  and  one  network  switch  operator.  Three  crewmembers  are  generally  available  to  assure 
continuous,  24-hour  operations.  ECS  operators,  under  the  direction  of  the  TCO,  initiate  manual 
target  engagements  or  enable  the  Patriot  to  engage  selected  types  of  targets  automatically.  The 
TCO  is  usually  an  air  defense  lieutenant  or  warrant  officer  and  the  TCA,  a  junior  or  mid-grade 
air  defense  noncommissioned  officer  (NCO).  While  AMD  headquarters  establishes  priorities  for 
protection,  alert  state,  weapons  system  status,  and  Rules  of  Engagement  (ROE)14  control  the  air 
defense  battle,  the  ultimate  FU  engagement  decision  is  made  by  the  TCO  and  crew  of  the  ECS 
using  information  available  to  them  at  the  time.  While  the  higher  headquarters  may  provide 
guidance  and  directives,  nothing  precludes  the  battery  from  taking  actions  in  self-defense  against 
air  or  missile  attacks. 

The  potential  threats  that  could  require  defense  by  Patriot  capabilities  are  numerous.  For 
example,  a  brief  and  unclassified  review  of  order  of  battle  and  organizational  factors  indicates 
that  over  30  other  nations  in  the  world  hold  operational  tactical  (shorter  than  300  kilometers  [km] 
in  range)  and  theater  ballistic  missile  (TBM)  systems  (George  C.  Marshall  &  Claremont 
Institutes,  2012a  &  2012b).  TBMs  vary  from  the  larger  intercontinental  ballistic  missiles  to 
smaller  tactical  systems.  Larger  TBMs  may  have  a  range  of  up  to  3,500km.  TBMs  are  designed 
to  carry  not  only  conventional  warheads  but  also  chemical,  nuclear,  conventional,  or  improved 
conventional  warheads.  Older  systems  are  retained  from  the  Cold  War  Era  by  some  nations; 
other  systems  are  the  product  of  more  current  development  and  fielding.  At  least  20  nations  hold 


14  ROE  are  rules  or  directives  to  military  forces  that  define  the  circumstances,  conditions,  limitations,  procedures, 
and  manner  in  which  the  use  of  force  can  be  initiated  or  continued.  For  example,  prior  to  hostilities,  ROE  may  be 
very  restrictive  to  prevent  the  engagement  of  a  potentially  unfriendly  aircraft  until  the  aircraft  commits  an  aggressive 
act  or  penetrates  into  a  friendly  territory  beyond  the  limits  possible  due  to  simple  navigation  error. 
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operational  Land  Attack  Cruise  Missile  (LACM)  systems.  Most  LACMs  have  variants  that  are 
both  surface  (land  or  sea)  and  air  launched.  A  few  models  can  be  submarine  launched.  LACMs 
also  vary  in  range  with  short-range  systems  reaching  out  to  300km  while  others  have  a  1,000km 
range.  They  also  vary  from  the  more  common  subsonic  systems  (similar  to  the  U.S.  BGM-109 
Tomahawk),  to  supersonic  systems  (some  held  by  China  [PRC]  or  Russia).  Additionally,  Russia 
and  India  are  jointly  developing  a  hypersonic  system  reported  to  travel  at  least  five  times  faster 
than  the  speed  of  sound  (Indian  Defense  Update,  2016).  LACMs  may  also  deliver  a  full  array  of 
warhead  types. 

Similarly,  over  100  nations  of  the  world  possess  inventories  of  aircraft  classified  as 
combat  aircraft  and  combat  helicopters  (Flight global,  2013).  These  air  forces  vary  in 
capabilities,  training,  and  platform  operational  availability.  Fixed  wing  combat  aircraft  include 
bombers,  fighters/interceptors,  fighter-bombers,  or  multi-purpose.  These  Air-Breathing  Threats 
(ABTs)  are  armed  with  a  wide  array  of  cannon  and/or  machine  gun  systems  and  may  be  purpose 
designed  or  modified  to  carry  air-to-surface  missiles  and  bombs  with  chemical,  nuclear, 
conventional,  or  improved  conventional  payloads.  Modem  ABTs  generally  travel  at  high 
speeds,  and  must  be  detected  and  reported  by  satellite,  airborne,  and/or  surface  based  radars  and 
sensor  systems.  Based  on  radar  signature,  ABTs  are  designated  as  friend  or  foe,  and  are 
accurately  tracked  for  successful  engagement.  As  an  example,  the  Korean  People’s  Army  Air 
Force  (North  Korea)  retains  more  than  570  combat  aircraft  and  150  combat  helicopters.  While 
some  of  their  aircraft  are  obsolete,  operational  availability  of  other  platforms  is  suspect,  and 
proficiency  training  in  some  units  is  minimal,  the  North  Korean  Air  Force  holds  modem  aircraft 
in  trained  front  line  priority  units.  By  today’s  standards.  North  Korea  poses  a  very  credible 
threat  to  U.S.  Forces  and  our  allies  in  the  region. 

TBMs  and  LACM  share  some  common  characteristics.  While  a  few  older  versions 
require  fixed  sites  for  launch,  most  modern  ground  launched  systems  are  fired  from  modified 
heavy  truck  or  tracked  carriers,  fully  integrated  transporter-erector-launchers.  Most  transporter- 
erector-launchers  can  be  readied  or  fueled  in  a  short  time  and  may  be  accompanied  by  support 
and  reload  vehicles,  dramatically  reducing  the  time  required  to  displace,  reload,  and  fire  again. 
Launch  locations  can  be  purpose  prepared  sites,  a  stretch  of  highway,  a  field,  or  an  accessible 
clearing  in  a  forest.  Most  have  improved  accuracy  designs  and  more  modern  systems  have 
integrated  counter  measures.  TBMs  have  short  flight  times  (medium  range  systems  are  +/-  seven 
minutes;  shorter  range  systems  are  +/-  160  seconds)  and  low  launch  and  flight  signatures  that 
complicate  detection,  tracking,  and  classification.  Reentry  speeds  for  some  long  range  TBMs 
approach  Mach  25,  but  atmospheric  friction  or  deliberate  deceleration  systems  slow  the  reentry 
vehicle  and  warhead  in  the  lower  atmosphere. 

Hostile  air  and  missile  attacks  would  seldom  come  from  a  single  system  or  platform.  Air 
offensive  operations  are  coordinated  efforts  employing  varied  types  of  combat  aircraft  and 
missile  systems  in  diverse  roles.  Generally  air  and  missile  attacks  are  massed  against  critical 
command  and  control  facilities,  fixed  installations,  specific  maneuver  units,  or  terrain  restricting 
the  maneuver  of  the  opposing  force.  Specialized  systems  may  be  targeted  against  air  and  air 
defense  systems.  The  effective  suppression  of  enemy  air  defense  (SEAD)  has  become  a  key 
aspect  for  winning  the  air  battle  in  modern  era  conflicts  (Department  of  the  Army,  2006).  Air 
defense  systems,  such  as  Patriot,  become  critical/high  payoff  targets  for  the  enemy  as  they  strive 
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to  gain  air  supremacy15  and  dominance  in  the  sky.  A  weapon  frequently  employed  in  SEAD 
operations  is  the  anti-radiation  missile  (ARM).  ARMs  are  specialized  air-to-surface  missiles 
designed  to  detect  and  home  in  on  an  active  radio  emission  source,  typically  an  opponent’s 
ground-based  air  defense  radar.  Tailored  SEAD  strike  packages  frequently  include  specialized 
ARM  launchers  as  well  as  jammers  and  escort  aircraft  armed  with  cluster  bombs  to  destroy  air 
defense  command  posts,  launchers,  and  support  equipment.  A  SEAD  strike  package  has  the 
objective  of  destroying  or  disrupting  one  or  more  of  the  opponent’s  air  defense  sites  and 
inflicting  sufficient  damage  to  keep  them  out  of  operation  for  an  extended  period. 


15  Air  supremacy  is  a  position  or  state  in  armed  conflict  where  a  side  holds  complete  control  of  air  warfare  and  air 
power  over  opposing  forces.  It  is  defined  by  NATO  and  the  United  States  Department  of  Defense  as  the  "degree  of 
air  superiority  wherein  the  opposing  air  force  is  incapable  of  effective  interference. 
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APPENDIX  B 

Example  Scenarios  and  Decision  Points 
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Summary  of  Cognitive  Skills  Training 


•  The  training  is  organized  into  three  progressively  more 
difficult  sections.  As  training  scenarios  become  more 
complex,  trainees  are  provided  less  direct  feedback  after  each 
decision,  prediction,  or  prioritizationthey  make.  Instead,  the 
consequences  of  their  decisions  are  allowed  to  play  out  and 
they  are  given  an  opportunity  to  try  again.  At  the  end  of  each 
section— includingmultiple  decision  points— trainees  are 
provided  feedback  summarizing  the  similarity  of  their 
responses  to  those  of  the  SMEsfor  that  section  of  the 
training. 

•  At  the  end  of  a  section,  auditory  feedback  is  provided  that 
includes  an  explanation  of  how  SMEs  arrived  at  their 
decisions,  predictions,  and  prioritizations.  This  feedback  also 
includes  suggestions  for  how  trainees  may  better  align  their 
cognitive  processes  with  those  of  the  SMEs. 
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Crawl  Scenario  Example 
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Crawl  Scenario  Part  I 


Routine  Traffic 

Situation  Summary; 

increase  m  routine  Nntfy  m  traftc  and  operator* 

ADW  Wflt 
Alert  Stata  2 

WEAPONS  HOLD 

Situation  Summary. 

Rury  ot  fnantfy  m  traftc  and  operator*  oontnua 

IBM  Track  (060)  eppeerv 

Track  *  travarang  your  AO.  not  mtvng  toward  tha 
protected  as  sat 


(1)  Routine  Traffic 


(2)  TBM  appears,  but  not 
moving  toward  protected 
asset 


(3)  Trainee  provided 
summary  of  situation. 


Decision  Reasons  for  Decision 


Place  an  X  beside  tha  two  (2)  Hants  that  best  represent  your 
decisions  in  this  situation  or  tha  actions  that  you  should  taka. 

□  Designate  the  track  tor  engagement  hold, 

□  Report  the  track  and  anomaly 

□  Request  ICC  verification  of  the  track. 

□  Set  TBM  A  tor  Auto  Engagement. 

□  Wait:  observe  the  track  lor  chargee. 


(4)  Trainees  asked  to  select  two 
items  best  representing  their 
decisions.  They  are  also  asked  to 
identify  three  items  of  information 
that  most  contributed  to  their 
decision(s).  Trainees  are  provided 
feedback  comparing  their  decisions 
and  rationale  to  those  of  the  SMEs. 


an  X  m  the  boxes  baatda  the  Im  (3)  ^formation  categories  or  flame  of 
information  mat  moat  contributed  to  the  totrrxaalicn  of  you  daemons  or  saiaci 

Mission; 

O  Nochangas 
t  :  ADW  WHITE 

□  Atari  Stala  2 

□  WEAPONS  MOU) 

Enemy: 

□  Track  data  lor  009 

Terrain  and  Weather: 

N/A 

Troops  and  Equipment 

□  System  Status  Green  Bar 


Launcnec/MrtMe  Status  (PAC2  «/PAC3  32) 

1  Green 

2  Green  3  Green 

4  Green 

RAC3  X  16 

PAC2X4  PAC3X16 

PAC2X4 

Tima: 

WA 
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Crawl  Scenario  Part  II 


An  Unknown  Aircraft  Appears 

Situabon  Summary: 

Routine  tnarxjy  ar  traflfc  and  operabone  contna 

An  unexpected  and  u*no*n  track  <007)  appear* 
mo%mg  South. 


■  (5)  Unknown  aircraft  moving  south 
toward  protected  asset. 


(6)  Trainee  provided  summary  of 
situation  concerning  unknown 
track. 


Decision  Reasons  for  Decision 


Decisions  for  the  Unknown  Track 

Race  an  X  besxte  the  two  (7)  iama  N(  beat  rapreearc  you 
deoeona  r>  tNs  ntuat  on  cr  the  actons  that  you  should  take 

□  Designate  Track  007  at  a  hoette 
f  Oaegnaba  Track  007  tor  manual  angagsmant 
[  Request  ICC  vanieabon  o»  the  Track  007 
D  Prepare  to  angaga  Track  0Q7  aa  a  ahooMook shoot  target 
[  Wat  cPaarva  ihe  Track  007  tor  cfiangaa 


(7)  Trainee  selects  two  items  that 
best  representtheir  decisions, 
and  then  identify  three  items  of 
information  that  most 
contributed  to  their  decision. 
Trainees  are  provided  feedback 
comparingtheir  decisions  and 
rationale  to  those  of  the  SMEs. 


Race  an  X  m  the  boxes  baa  da  the  three  (3)  nkxmebcn  cetegonea  or  Mama  ot 
rkrmaton  that  moat  cooatidad  to  tha  tonndakon  ot  you  daemons  or  sateebona 

Mission 

□  No  changaa 
(  ADWWHTTE 

□  Atari  Stala  2 

l  WEAPONS  HOLD 

Enemy. 

□  Track  data  tor  09  7 

Terrain  and  Weather 
Utk 

Troop  a  and  Equipment 

□  System  Status  Graan  Bar 
p  kAsata. launcher  Status 

Tima: 

H/A 
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Crawl  Scenario  Part  III 


Three  p)  U/l  Aircraft  on  Scope 

Situation  Summary. 

NawAOOe 
*OW  YBJ.OW 

Atari  State  2 

WEAPONS  MOLD 

Tt*aa  u*nw»n  arc rat  miaar  an  sooca  Syrg  n 
tele*  c  a  oMi  n  Nardknd  anpaca 

1  X  naar  M  No^terd  Suttend  twda.  MNW 

2XvmteNanto  Pwwnoal** 


(8)  Three  unknown  aircraft  flying  in 
tight  elliptical  orbit  at  periphery  of 
scope. 


(9)  Trainee  provided  summary  of 
situation  concerning  the  3 
unidentified  aircraft. 


Decision  Reasons  for  Decision 


(10)  Trainee  selects  two  items  that  best  representtheir  decisions,  and  to 
identify  what  the  U/l  tracks  could  possibly  be.  Then,  they  are  asked  to 
identify  three  items  of  information  that  most  contributed  to  their  decision 
Trainees  are  provided  an  overall  rating  of  how  closely  their  decisions  and 
selections  of  critical  information  matched  the  experienced  SMEs' 
responses.  Additional  auditory  feedback  is  provided  to  assistTrainees  in 
better  understanding  and  aligning  their  cognitive  processes  with  those  of 
SMEs. 


Decisions  for  the  3  U/l  Aircraft 


Plaoa  an  X  bwda  tha  two  (2)  Hama  tut  bast  rapraaant  you 
daemon*  in  tha  situation  or  lha  actions  that  you  staid  taka 

□  Qata*y  thu  t trmi  unknown  track*  as  hot!** 

□  Dategnata  tw  tracks  tor  angaqamant 

[  Raqjast  ICC  vanlcaaon  of  tracks  and  arcnll  typa. 
Wtet.  obaarva  tha  fratks  lor  changt* 


What  typa  ABTs  do  tot**  tracks  Ikaty  rapraaanf? 
Placa  an  X  in  tha  appropnata  box 

□  UAVInconnamnc*) 

□  UdVfSEAD) 

D  MannM  nonwiinM  arcraR 
O  Mamad  itnaa  aroaft 
P  Ra mpoaatola to daramwta 


Placa  an  X  in  m*  bom  bated*  tha  Vvaa  (3)  Mormaton  calagonas  or  Sam*  of 
nSonrteaon  that  mot  contnbuiad  to  t>a  tormjubon  of  yotr  daoacna  or  Mkcton. 

Mhslons 

□  Mo  changas 

P  AOWYQXCW 

□  Atari  Stata  2 

D  WEAPONS  MOLD 

Enamy 

□  Tracks  dais  tor  Tracks  103. 107.  and  100 
FF:  No  Raaponsa  from  afl  Tracks 

Tarratn  and  Waathar 
WK 

Troops  and  Equipmant 

□  Syxtam  Status  Qraan  Bar 
D  Lsncrw/ttetea  Status 

Tima; 

WA 
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Walk  Scenario  Example 
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WalkScenario  Parti 


(1)  Increased  alert.  Hostile 
aircraft  on  scope,  including  a 
jammer  and  slow  flying  air- 
breathing  threat  (ABT)  in 
elliptical  orbit.  ABTs 
approaching  border;  coalition 
aircraft  scrambled. 


Massed  ABTs  Moving  Southwest 

' 

Situation  Update: 

ADAFCO  (ftssemmates. 

\  «  / 

AWD  RED 

\  o  e  / 

Alert  State  1 

WEAPONS  HOLD  remains  to  effect 

\  a  / 

A  messed  tght  of  ABTs  cross  the  border  end  head  SSW 

toward  your  protected  asset 

Multiple  Aircraft  Attacking 

<6  „  7 

Situation  Update: 

V  7 

The  famrrm  breaks  orbit  headmg  WSW 

\  \  V  ) 

\  cf~K'  / 

Coe* Sen  Rgh furs  maneuver  to  intercept  the  nbound 

\  °-  Q  / 

hostie  aircraft  m  the  NNE  of  you  scope 

A  second  light  of  hoetle  ar  craft  snler  your  scope  from 

the  NfW  They  are  movsig  r aptefry  toward  your  protected 

\  ^  / 

asset 

(2)  Massed  ABTs  moving 
southwest  toward 
protected  asset. 


(3)  Jammer  begins 
moving  west-southwest. 
Coalition  aircraft  move 
to  intercept  hostile 
aircraft  in  north- 
northeast  of  scope. 
Second  flight  of  hostiles 
enter  scope  from  north- 
northwest  moving 
toward  protected  asset. 


(4)  Trainee  provided  page  summarizing  the  situation.  Then  asked  to 
prioritize  their  top  three  decisions  and  rationale. 


Threat  Priorities 


Prioritize  the  top  three  threats  (1  -3)  to  your  mission  based  on  the 
current  situation.  Note,  t  is  the  greatest  threat.  Place  a  number 
in  the  box. 

□  Track  120 

0  Tracks  120.  121,  and  122 
■  Track  117 
B  Track  113 


(5)  Trainees  are  provided  feedback  on  their 
decision  priorities  and  rationale. 


Reasons  for  Priorities 


Dol  ermine  the  Wn  (3)  elements  of  information  or  data  that  contnbutod 
moat  to  tha  formulation  of  yoor  priority  dadaao ns.  Plaos  an  X  in  trio 
boxes  beside  tha  item 

Mission: 

□  No  changes;  attack  underway 

□  ADW  RED 
D  Atari  State:  1 

□  WEAPONS  HOLD 

Enemy: 

□  Track  Data  lor  Track  129:  IFF:  No  Response 

□  Track  Data  tor  Tracks  120, 121.  end  122:  IFF:  No  Response 

□  Track  Data  lor  Track  117:  IFF:  No  Response 

□  Track  Data  lor  Track  113:  IFF:  No  Response 
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WalkScenario  Part  II 


The  Attack  Develops 

Situation  Updata 

(% 

CxM**on  V/r.srs  *-o  tfWfvn 

V? 

o 

Enamy  arc raft  are  4 sparser]  Ian  th«r  bU*  kM-xJstcru 

and  sustan  mrttai  loaaaa 

0~;  . 

Tha  tarrrrm  convuM  but  tjraaa*  c»t»t  mcwng 

to  tha  WSW  no  alWct 

Tha  ttoond  hot  Ua  ftg N  n  fa  NtM  o 1  you  acoca 
txrtrua*  to  acvroaui  you  proeactad  ««•( 

\  ^ 

(6)  The  attack  develops. 
Enemy  aircraft  are  dispersed. 
Jammer  is  operating,  but  with 
no  effect.  Second  flight 
continues  to  approach  asset 
from  NNW. 


Coalition  Fighters  Interdict  an  Attack 

Situabon  Updato: 

Th#  AST  fcrmabon  massed  n  tha  East  of  you  scope 
sustains  additional  losses  after  continued  engagement  by 
coaJit)cn  Sghlers 

The  second  hosts*  flight  in  the  west  of  your  scope 
continuee  to  approach  you  protected  asset. 

The  yammer  assures  an  orbit  but  has  no  eflect 


(7)  ABT  formation  sustain 
additional  losses.  Second 
hostile  flight  continues 
approach.  Jammer 
assumes  orbit,  but  no 
effect. 


(8)  Trainees  are  provided  a  summary  of  the  situation.  Then  they  are 
asked  to  determine  the  probability  of  occurrence  for  track  split,  missile 
launch,  and  strafe/bomb  targets  in  the  protected  area  for  each  track. 

Probability  Ratings  Reasons  for  Ratings 


Determine  the  probability  of  occurrence  (or  each  track  or  track 
set  in  the  table  below.  Place  a  number  In  the  boxes  below: 

1  =  Highly  probable.  2  =  Possible.  3  =  Improbable 


Hostile 

Tracks 

Track  Split 

Launch  ARAM 

Strafe/Bomb 
Targets  in 

AA  Dog 

121.122.131. 
132.  135 

3 

2 

1 

129(3) 

1 

2 

1 

,13 

3 

3 

3 

(9)  Trainees  are  provided  feedback 
on  their  probability  ratings  and 
rationale. 


Determine  (he  three  (3)  elements  of  information  or  data  that  contrtbutad 
moat  to  ttie  formulation  of  your  probaMrty  decisions  Place  an  X  In  the 
boxes  beside  the  item. 

Mission: 

O  Factors  and  Alerts 
Enemy: 

□  Track  Data  tor  Track  121. 122, 131, 132,  and  135 

□  Track  Data  for  Tracks  129 
Q  Track  Data  tor  Track  117 

□  Track  Data  for  Tracks  113 

Terrain  and  Weather: 

N/A 


Troops  and  Equipment: 

□  System  Status:  Green  Bar 


□ 

launcher/Mtssile  Status  (PAC2  8/PAC3  32) 

1  Groon 

2  Green 

3  Gr«n 

4  Green 

PAC3X  16 

PAC2X4 

j  PAC3X16 

PAC2X4 
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Timo: 

N/A 


WalkScenario  Part  III 


0 


W 

* 

CO  o 
Q> 


© 


V 


(9)  Four  SCUDs  on  scope, 
headed  toward  AA  Dog. 

Hostile  flight  in  west  continues 
to  close  on  the  protected  area. 
ABT  formation  appears  to  be 
moving  to  exfiltrate.  Jammer 
continues  to  orbit. 


( 10)  Directed  to  engage 
SCUDs.  System  alerts  an 
air  launch. 


(11)  Trainees  are  provided  a  summary  of  the  situation.  Then  they  are  asked  to 
prioritize  threats  to  their  mission,  IDthe  theater  ballistic  missiles,  and  IDthe 
information  that  contributed  to  their  decision.  ( 12)  They  are  then  provided 
feedback  on  their  responses. 

Decisions  Contributing  Information 


Prioritize  the  top  three  threats  (1  -3)  to  your  mission  based  on 
current  situation.  Note.  1  is  the  greatest  threat. 

W  Tracks  137,  138,  139,  and  140 
II  Track  142 
|  Track  129 
Track  113 


Indicate  the  most  probable  classification  of  the  Inbound  TBMs 
(Tracks  137,  138,  139,  &  140).  Place  an  X  in  the  appropriate  box. 

□  SS-21  Scarab  B 

□  Hwasong  5  (SCUD  B  variant) 

□  Hwasong  6  (SCUD  C  variant) 

□  No  Dong  1/No  Dong  A 

Indicate  the  type  of  ARM  or  ASM  indicated  by  the  inbound 
Track  142.  Place  an  X  in  the  appropriate  box. 

□  C-802KD.  KD-88  (CSS-N-8  Saccade)  •  ASM 

□  Kh-23  (AS -7  Kerry)  ■  ASM 

□  Kh-58EM  (AS- 1 1  Kilter)  ■  ARM 
C  Kh-31P  (AS-17  Krypton)  -  ARM 


Determine  the  three  (3)  elements  of  information  or  data  that  contributed 
to  the  formulation  of  yotx  threat  pnonty  decision.  Race  an  X  h  the 
boxes  beside  the  Item. 

Situation  Update: 

□  ADWREO 
G  Alert  State:  1 

D  WEAPONS  HOLD 

Enemy: 

I  Track  Data  lor  Tracks  t37.  138,  139.  and  140;  IFF:  No  Response 
D  Track  Data  for  Track  129;  IFF:  No  Response 
(  Track  Data  lor  Track  142;  IFF:  No  Response 
G  Track  Data  for  Track  113;  IFF:  No  Response 

Terrain  and  Weather: 

MCA 

Troops  and  Equipment 
G  System  Status:  Groan  Bar 

□ 


Launchior/Mtssile  Status  (PAC2  8/PAC3  32) 

1  Green 

2  Green 

3  Green 

4  Green 

RAC3X  16 

PAC2  X  4 

PAC3X  16 

RAC2  X4 
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WalkScenario  Part  IV 


SCUDs  on  Scope  1 

Situabon  Update 

AOAf  CO  warns  of  TBM  Ultum.  lot#  (4)  SCUO*  appear 
on  your  scop*  corvtrgrg  cn  AA  DOG 

The  holla  ftgN  rt  tho  Waal  ccnlnjM  to  rapxP/  doea  on 
AAOOC 

Th*  ABT  formation  m»M  n  th#  Eaat  of  yax  acop« 
auttara  KkMcml  toataa  and  appears  to  t»  man*A*mg 
toaodhrata. 

Tha  jarrrrar  (1 1 7)  cononuaa  n  catJrt 


(13)  SCUDs  and  inbound  ARM 
have  been  destroyed.  Track 
split  and  air  launch  warning. 
Hazard  warning  on  Launch 
Station  3. 


(14)  Trainees  are  provided  a  summary  of  the  situation.  Then  they  are  asked  to  prioritize 
threats  to  their  mission,  ID  the  inbound  ARM/ASM,  and  ID  the  slow  moving  ABT,  and  to 
recommend  the  priority  action  for  the  Battery.  They  then  identify  information  that 
contributed  to  their  decision. 


Decisions 


Prioritize  the  threats  (1-3)  to  your  mission  based 
on  the  current  situation.  Note.  1  is  the  greatest 

threat. 

Indicate  the  classification/type  of  the  inbound  stow 
moving  ABT  indicated  by  Track  1 13.  Place  an  X  in 
the  appropriate  box. 

Q  Tracks  143(JAMMER),  144.  and  129  (ACR) 

H  Track  145 

B  Track  113 

□  Helicopter 

□  ASN-105B  UAV 

□  Shmel-1  UAV 

C  Harpy  UCAV 

Indicate  the  type  of  missile  indicated  by  the 
inbound  ARM/ASM  Track  145.  Place  an  X  in 

the  appropriate  box. 

What  is  your  recommended  priority  for  the  Battery? 
Place  an  X  in  the  box  with  the  highest  priority. 

□  C-802KD,  KD-88  (CSS-N-8  Saccade)  - 

ASM 

□  Kh-23  (AS- 7  Kerry)  -  ASM 

□  Kh-58EM  (AS-1 1  Kilter)  -  ARM 

F  Kh-31P  (AS- 17  Krypton)  -  ARM 

□  Reload  LSI 

□  Reload  LS  3 

F  Address  the  HAZARD  on  LS  3 

(15)  Trainees  are  provided  an  Contri  bliti  ng 

overall  rating  of  how  closely  Information 

their  decisions  and  selections  - 

of  critical  information  matched 
the  experienced  SMEs’ 
responses.  Additional  auditory 
feedback  is  provided  to  assist 
trainees  in  better 
understanding  and  aligning 
their  cognitive  processes  with 
those  of  SMEs. 


Determine  the  three  (3)  elements  ot  information  or  data  that 
contributed  most  to  the  formulation  of  your  threat  priority 
decision.  Place  an  X  in  the  boxes  beside  the  three  items. 

Mission 

□  ADW  RED 

□  Alert  State:  1 

□  WEAPONS  HOLD 

Enemy 

n  Track  Data  for  Tracks  143.  144,  and  129;  IFF:  No  Response 
n  Track  Data  for  Track  145;  FF:  No  Response 

□  Track  Data  for  Track  113;  FF:  No  Response 

Terrain  and  Weather 
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Run  Scenario  Example 
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Run  Scenario  I 


Run  Phase  Mission  1:  Battle  Stations 

Situation  Summary: 

ECS  Crows  ore  catod  to  battle  stations. 

ADW  YELLOW 
Alert  State.  2 
WEAPONS  HOLD 

JTF  526  has  indcabons  of  immnont  NortSand  a u  and 
nmsale  attacks  targeted  aganet  U  S  and  cotfbon  air  and 
air  defense  forces  ABT  launches  m  NOfltffld  have  stxged 
and  TBM  forces  are  on  alert  and  at  a  state  of  launch 
readiness. 

AOAFCO  provides  warning  ol  hostile  aircraft  approechng 
from  the  North  and  annooces  that  JTF  and  ooeftoon 
flQhter  aircraft  have  been  scrambled. 


(1)  Crews  called  to  battle 
stations.  Attack 
imminent.  Warning  of 
hostile  aircraft 
approaching.  Coalition 
aircraft  scrambled. 


ABTs  on  Scope 

Situation  Update: 


AOAFCO  0 
ADW  RED 
Alert  State:  1 

WEAPONS  HOLD  remains  in  effect 

You  are  drected  to  engage  TBMa  targeted  against  AA 
DOG  and  Missio  Engagement  Modes  h eve  been  updated 
to  AUTO. 

Three  (3)  hostie  fights  appear  on  your  scope  adranong 
toward  AA  DOG 

Coalilfon  fighters  appear  on  scope. 


(2)  ABTs  and  coalition 
fighters  on  scope. 
Directed  to  target  TBMs 
targeted  at  AA  Dog. 


Throat  Aircraft  Initiate  ECM 

Situation  Update: 

Inboard  hostie  aircraft  nttrted  ECM. 

CoaAbon  Alters  ma neuter  to  ntercept  hostile  aroraft. 


\® 


(3)  Threat  aircraft  initiate 
electronic 
countermeasures. 
Coalition  fighters 
maneuver  to  intercept. 


(4)  Trainees  are  provided  a  summary  of  the  situation. 


CONTINUED  ON  FOLLOWING  PAGE 
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Run  Scenario  I 


Decisions 


Prionttze  the  threats  (1  -  3)  to  your  mission  based  on  the  current 
situation.  Note,  1  is  the  greatest  threat.  Place  the  number  in  the 
appropriate  box. 

■  Inbound  ABTs 

|  Jammers  and  ECM 

■  TBMs 

Which  action  shotid  be  your  highest  priority  at  this  stage  of  the  air 
battle?  Place  an  X  in  the  box  beside  the  appropriate  item. 

□  Prepare  to  engage  ABTs. 

□  Prepare  to  engage  the  Inbound  tammers  only. 

□  Implement  ail  available  PATRIOT  Electronic  Counter 
Countermeasures  (ECCM). 


(4)  Trainees  are  asked  to 
prioritize  threats  to  their 
mission,  and  then  to 
identify  the  highest 
priority  at  this  point  in 
the  air  battle. 


Contributing  Information 


Determine  the  three  (3)  elements  of  Information  that  contributed  most  to  the  formula  bon  of 
your  priority  decision.  Place  an  X  In  the  boxes  beside  your  selected  items. 

Mission: 

□  No  change:  a»  attacks  are  underway  and  missile  attack  warnings  have  been  issued, 
targets  are  US/ooaWon  air  and  air  defense  forces. 

□  ADW  RED 

□  Alert  State:  1 

□  WEAPONS  HOLD 

Enemy: 

□  Data  for  Tracks  041  and  043;  IFF:  No  Response 

□  Data  for  Tracks  045  and  046;  IFF:  No  Response 

□  Data  for  Tracks  042  and  044;  IFF:  No  Response 

Terrain  and  Weather: 

NTA 

Troops  and  Equipment: 

□  System  Status  :  Green  Bar 


Launcher/Misstfe  Status  (PAC2  8/PAC3  32) 

1  Green 

2  Green 

3  Green 

4  Green 

PAC3X  16 

PAC2  X  4 

PAC2  X  4 

Time: 

NTA 


(5)  NO  DIRECT  FEEDBACK  IS  PROVIDED. 
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Positive  Outcome:  Decision  to  I  Negative  Outcome:  Decision  to 

Employ  ECCM  I  Engage  ABTsor  Jammers 


Run  Scenario  I  (Continued) 


The  next  step  in  the  scenario  is  based  on  how 
trainees  answered  the  previous  questions. 


Effective  ECM 

au 

Situation  Update: 

*  \  „ 

*•  “  ■ 

ECM  effectiveness  increases  as  the  jammers  come  closer. 

ADAFCO  disseminates  that  SCUO  launches  have  (seen 

detected. 

Coalition  aircraft  begin  engagements. 

(6a)  Enemy  ECM  are 
effective.  SCUD 
launches  detected. 
Coalition  aircraft 
begin  engagements. 


SCUDs  on  Scope  &  an  Air  Launch  Warning 

(7a)  Continued  ECM 

Situation  Updato: 

effects.  SCUDs  on 

Effective  ECM  continues. 

«  £  5 

scope.  Air  launch 

detected. 

SCUO  missiles  appear  on  scope. 

\  a  / 

An  Air  Launch  is  detected. 

ECM  Effects  Reduced 

Situation  Update: 

Jamming  effectiveness  has  diminished  with 

\  .RLfc/ 

W  e  c .  £ny 

Implementation  of  ECCM  measures. 

\  CJ  / 

ADAFCO  disseminates  that  SCUD  launches  have  been 

detected. 

Coalition  aircraft  begin  engagements. 

(6b)  Jamming 
effectiveness 
diminished.  SCUD 
launches  detected. 
Coalition  aircraft 
begin  engagements 


SCUDs  on  Scope  &  an  Air  Launch  Warning 

Situation  Updato: 

ECCM  has  localized  the  effectiveness  ol  jamming  against 

°6 

your  radar. 

t  / 

SCUO  missies  appear  on  scope. 

p 

An  Air  Lainch  is  detected. 

(7b)  ECCM  localized 
effect  of  jamming 
against  radar.  SCUD 
missiles  on  scope. 
Air  launch  detected. 
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Run  Scenario  I  (Continued) 

(8)  Trainees  are  provided  a  summary  of  the  situation. 


Air  and  TBM  Attacks  Are  Underway 

Mission: 

Enomy: 

No  chang*»:  **  *nd  mtsale  attacks  ara  underway 

Tracks  047. 062.  063.  and  054  ara  dassiftad  as  hostile 

Tracks  048.  049.  050.  and  051  ara  dassrSed  as 

ADW  RED 

TBMs 

a  hoe  No  TBMs 

Alert  State:  1 

GJPs  Southern  araa  of  AA  DOG  and  your  battery 

GPs:  Southern  area  of  AA  DOG  and  your 

WFAPONSMOLD 

location 

battery  location 

FF:  No  Response 

FF:  No  Response 

Torrain  and  Woathar. 

Track  Data: 

Track  Data: 

N/A 

Maadng  S  (168*)  toward  tha  protected  assot 

Headng  S  (193*)  toward  tha  protected  assot 

Slaap  <*re  angle  from  extromo  aftitudo 

Sleep  <*vo  angto  *om  extreme  tftrtudo 

Troops  and  Equipmont 

Spaad:  6.800  kV  3.500  m/s 

Speed  6.800  kV  3.500  m/s 

Systam  Status:  Graan  Bar 

Port  o(  ongrv  BMOA  10 

Port  of  ongrv  BMOA  8 

Laincher/MissAe  Status  (PAC2  8/PAC  3  32 

Tracks  041  and  043  (AC«)  ara  dasteted  as  a  hoatee 

1  Graan  2  Graan  3  Graan  4  Graan 

A8Ts 

Track  056  is  dassited  as  a  Mostte  ASM 

FF:  No  Response 

(potential  ARM) 

RAC3X16  PAC2X4  a  .  h  F*C2X4 

Track  Date: 

FF:  No  Response 

Maadng  SSE  (ISO*)  toward  protected  asset 

Track  Oata: 

AAAude  48  5  kft  14  8  km.  steady 

Headng  SSE  (150*)  toward  the  protected 

Tima 

Speed:  425  kt/ 2 10  m/s 

asset 

N/A 

Port  of  ongt*  Roach 

Altitude  46  kJV  14  km,  dMng 

Length  16  8  meters 

Spaed:  1 .470  kt/  780  m/s 

Port  of  ongrv  Track  043 

Length:  5  meters 

Decision 


Prioritize  trie  top  three  threats  (1  -  3)  to  your  mission  based  on  the 
current  situation.  Note,  t  is  the  greatest  threat.  Place  the  numbers 
In  the  boxes  beside  the  item. 


■  Tracks  047.  048.  049.  050,  061.  062.  053,  and  054 
H  Tracks  041  and  043 

■  Track  056 


Contributing  Information 


Determine  the  iyn  (3)  items  or  alruntt  of  information  or  deployed  data  that  ccntrtiuied 
most  to  the  formulation  of  your  threat  priority  daemon  Race  an  X  In  the  boxes  beside  the 


Mission: 

□  No  changs;  ar  and  msato  attacks  on  underway. 

□  ACW  RED 

□  Mart  Stata:  1 

□  WEAPONS  MOLD 

Enemy: 

□  Data  for  Tracks  047.  062,  OS3.  and  064;  FF:  No  Reaper*# 

□  Oata  for  Tracks  040.  049.  060.  and  061;  fF  No  Raaponaa 
O  Data  for  Tracks  041  and  043;  Iff:  No  Response 

□  Data  for  Track  066;  FF  No  Ftesponsa 

Torrain  and  Weather: 

N/A 

Troops  and  Equlpmant: 

□  System  Status  :  Graan  Bar 


Launcher/Meate  Status  (PAC2  8/PAC3  32) 

1  Green 

2  Green 

3  Groan  4  Groan 

PAC3  X  16 

PAC2X4 

PAC2X4 

Tima: 

N/A 


(9)  NO  DIRECT  FEEDBACK  PROVIDED. 
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Positive  Outcome:  I  Negative  Outcome: 

Decision  to  Engage  ARM  I  Decision  to  Engage  SCUDs 


Run  Scenario  I  (Continued) 

The  next  step  in  the  scenario  is  based  on  how 
trainees  answered  the  previous  questions. 


SCUDs  Engaged  as  the  Priority  Threat 

Situation  Update: 

\  fie-  Jc"  / 

The  inbound  SCUDS  were  engaged  as  the  priority  threat. 

\&>  y 

The  Inbound  AS-1 1  Kilter  ARM  is  seconds  from  impact. 

H®-  /  - 

* 

(10a)  SCUDs 
engaged.  ARM  is 
about  to  impact. 


ARM  Destroyed:  SCUDs  Targeted 

~7 

Situation  Update: 

\  o  -  O'  / 

Missile  Engagement  Modes  were  updated. 

/,#>  y 

The  inbound  ARM  was  destroyed. 

\®  / 

SCUDs  have  been  designated  for  automatic  engagement. 

\*  / 

(10b)  ARM 
destroyed.  SCUDs 
designated  for 
engagement. 


SCUDs  Engaged 

(11)  SCUDs 

Situation  Update: 

\  b  oD  J3:  / 

destroyed.  New 

\  ©  3/ 

threat  from  eastern 

The  inbound  SCUDS  are  being  engaged.  PATRIOT  is 

\  o  aJ 

\  tr  7 

flank;  reorient  35° 

effective. 

\Vvi  / 

Mission  modification;  new  threat  from  the  Eastern  (tank. 

Reorient  to  35°. 
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Run  Scenario  I  (Continued) 


Sunrise,  a  New  Azimuth,  and  a  Fault 

b 

(12)  New  azimuth. 

Situation  Update: 

d  / 

\  V 

Fault  detected. 

The  radar  has  been  aligned  to  the  new  azimuth. 

The  system  begins  to  radiate  and  processors  are  catching 
up  on  track  classifications. 

\  W 

\  a  / 

\  / 

A  tault  displays  in  the  Alert  Line. 

Bogies  on  the  Right  Flank 

Situation  Update: 

Enemy  aircraft  approach  down  the  right  (lank. 
The  taut  remains  on  the  Alert  Line. 


(13)  Enemy  aircraft 
on  right  flank.  Fault 
detected. 


(14)  Trainee  is  provided  a  summary  of  the  situation. 


Decision 


Contributing  Information 


Prioritize  your  top  two  priority  recommendations  to  the  battery  (or 

Determine  Vw  three  (3)  etements  of  n  formation  that  ccntrtxXed  most  to  the  formOation  of 

maintenance  and  reload  eriorts  (1-2)  based  on  trie  current 

your  priority  decision  Ptaoa  an  X  m  tha  bow  beekto  your  i  afoot  ed  Kama. 

Situation.  Note,  1  is  the  highest  prionty.  Ptace  the  number  in  the 

Mission: 

bo*  beside  the  item. 

□  No  changes:  AA  DOG  remains  me  primary  protected  asset,  tedar  la  to  reonent  to  35': 

threat  anticipated  from  BMOAs  3.  7.  and  9 

□  ADWRED 

1  Prepare  to  conduct  reload  operations  on  LS  t  and  LS  3 

□  Alert  Stele:  t 

■  Reload  LS  1 

□  WEAPONS  HOLD 

■  Reload  LS  3 

|  Address  the  Hazard  condrtion  on  LS  2 

Enemy: 

□  Track  Data  tor  Tracks  008  and  OOD.  FF:  No  Raaponaa 

□  Track  Data  la  Tracks  010.  Oil.  012.  and  013.  FF:  No  Response 

Terrain  and  Woathor. 

(15)  Trainee  are  asked  to 

N/A 

prioritize  recommendations  to 

Troops  and  Equipment: 

the  Battery  for  maintenance  and 

□  System  Status  :  Amber 

reload  efforts.  Then,  they  are 

□  Launcher /Mtssrie  Status  (PAC2  S/PAC3  14) 

1  Green  3  Green  4  Green 

asked  to  identify  what 

PAC3X10  P3IBI  PAC3X4  PAC2  X  4 

information  they  relied  on  to 

Time: 

determine  this  prioritization. 

N/A 
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Run  Scenario  I  (Conclusion) 


Mission  3A  AAR 

Name 

<lastName> 


Rank 


<rank> 


How  did  your  initial 

Stfnrtar 

X 


decisions  compare  to  those  of  PATRIOT  SMEs? 

Related  Dissenter 


Date 

<date> 

How  dKj  your  consideration  of  data  elements  and  information 
compare  to  considerations  of  PATRIOT  SMEs? 


Mission 

Factors 

Enomy 

IFF 

Track  Data 


Return  to  the  Mss»on 
Skits  Tranng  Man  Menu 


Terrain  and  Weather 

Troops  and  Equipment 
System  Status 
Lauxtor /Missile  Status 


Knomatics 

Tima 

Similar 

Related 

Dusimlar 

X 

(16)  Trainees  are  provided  an  overall  rating  of  how  closely  their  decisions 
and  selections  of  critical  information  matched  the  experienced  SMEs' 
responses.  Additional  auditory  feedback  is  provided  to  assistthem  in 
better  understanding  and  aligning  their  cognitive  processes  with  those  of 
SMEs. 
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APPENDIX  C 

Pre-Training  Demographic  and  Training  Survey 


C-l 


Pre-Training  Demographic  and  Training  Survey  (v3) 


1.  Rank:  _  2.  (WO)MOS: _ 

3.  Current  assignment  and  duty  position:  (i.e.  TCO,  TCA,  etc. 

4.  How  long  (months): _ 


5.  Using  Table  2-1  below,  please  circle  O  the  highest  Patriot  Gunnery  Table  that  you  have  been 
externally  evaluated  as  having  attained  and  the  recital  aim  level  (RAL)  that  you  have  attained. 


Level 

Table 

Subject  Matter 

Corresponding 

RAL 

Basic  Gunnery 
Tables 

1 

Basic  System  Skills 

II 

Ready-for-Action  Drills 

III 

Basic  Air  Battle  Management 

1,2,  3,4,5 

IV 

Basic  Gunnery  Certification 

5 

Intermediate 
Gunnery  Tables 

V 

Air  Battle  Management/Missile  Reload 

6,  7,  8,  9 

VI 

Daytime  March  Order  and  Emplacement 

VII 

Precertrfication  Tables  V  and  VI 

1  through  10, 11 

VIII 

Intermediate-Level  Gunnery  Certification 

11 

Advanced 
Gunnery  Tables 

IX 

Advanced-Level  Air  Battle  Management  and  Missile 
Reload 

12, 13 

X 

Nighttime  MO&E  Under  Varying  CBRN  Conditions 

XI 

Precertification  Tables  IX  and  X 

14.  16,  17 

XII 

Advanced-Level  Gunnery  Certification 

17 

Table  2-1,  FM  3-01.86  Air  Defense  Artillery:  Patriot  Brigade  Gunnery  Program,  23  Dec  08 
w/CHl. 
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6.  Place  an  “X”  in  the  appropriate  box  for  each  question: 


Item 

Never 

Occasionally 

Often 

Always 

A 

The  Patriot  engagement  training  received 
in  my  unit  has  been  focused  on 
remembering  and  understanding 
procedures. 

B 

My  Patriot  engagement  training  has 
focused  on  applying  the  procedures  that  I 
can  recall  and  understand. 

My  Patriot  training  has  provided  information 
that  my  crew  and  I  were  required  to  analyze 
and  evaluate  information  and  data  in  order  to: 
(See  C  -  N  below) 

C 

Set  work  and  task  priorities 

D 

Review  and  modify  priorities 

E 

Set  engagement  priorities  for  tracks  or 
targets  on  my  scope 

Anticipate: 

F 

Track  separations 

G 

Launch  of  ASMs  or  ARAMs 

H 

Jamming 

I 

The  intent  of  a  track  or  a  hostile 
formation 

J 

Launcher  reload  requirements 

K 

Need  to  alter  system  settings  on  the 

Patriot 

L 

The  probable  sequence  of  events  during  a 
hostile  attack 

Act  to  counter: 

M 

Erroneous  classification  of  tracks 
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N 

Coordinated  SEAD  directed  against  my 
battery 

7.  I  have  been  provided  models  or  methods  for  problem  solving  and  engagement  prob 

lems: 

YES: _  NO: 


8.  I  have  been  provided  scenarios  and  situations  to  assess  my  critical  thinking  and  problem 
solving  and  given  feedback. 

YES: _  NO: _ 

9.  I  have  been  trained  in  situations  to  practice  my  critical  thinking  and  problem  solving  skills. 

YES: _  NO: _ 
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APPENDIX  D 

Results  of  the  In-Route  Training  Assessment 
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In  the  following  Tables  D-l  through  D-3,  Soldiers’  responses  to  the  training  scenarios  are 
reported.  The  highlighted  responses  indicate  the  SME  consensus  responses  against  which 
individual  trainees’  responses  were  scored. 

Table  D-l 

Crawl  Phase  Training  Module  In-Route  Assessment  Results 


Module 

Decision/F  actors 

n  of  20 

Decisions  for  the  TBM  Track 

Decision 

Designate  the  track  for  engagement  hold 

H 

Report  the  track  and  anomaly 

11 

Request  ICC  verification  of  the  track 

9 

Set  TBM  A  for  auto  engagement 

1 

Wait;  observe  the  track  for  changes 

4 

Factors 

Mission: 

No  Change 

1 

ADW  WHITE 

B 

Alert  State  2 

3 

WEAPONS  HOLD 

6 

Enemy: 

Track  Data  for  089 

15 

Troops: 

System  Status:  Green  Bar 

2 

Launcher/Missile  Status:  All  Green 

2 

Decisions  for  an  Unknown  Track 

Decision 

Designate  track  097  as  a  hostile 

2 

Designate  track  097  for  manual  engagement 

2 

Request  ICC  verification  of  the  track  097 

18 

Prepare  to  engage  Track  097  as  a  shoot-look-shoot  target 

3 

Wait;  observe  track  097  for  changes 

13 

Factors 
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Mission: 

No  changes 


8 


ADW  WHITE  14 

Alert  state  2  1 

WEAPONS  HOLD  | 

Enemy: 

Track  data  for  097  17 

Troops  and  Equipment: 

System  Status:  Green  Bar  0 

Missile/Launcher  Status:  All  Green  1 


Decisions  for  the  3  Unidentified 
Aircraft 

Decision: 

Classify  the  three  unknown  tracks  as  hostile 
Designate  the  tracks  for  engagement 
Request  ICC  verification  of  tracks  and  aircraft  type 
Wait;  observe  the  tracks  for  changes 

UAV  (Reconnaissance) 

Type  of  Aircraft: 

UCAV  (SEAD) 

Manned  reconnaissance  aircraft 

Manned  strike  aircraft 

It  is  impossible  to  determine 


1 

1 


9 

8 

1 

1 

0 


Factors: 

Mission: 

No  changes  2 

ADW  Yellow  15 

Alert  State  2  0 

WEAPONS  HOLD  4 

Enemy: 

Tracks  data  for  Tracks  103,  107,  and  109  lf| 

IFF:  No  response  from  all  tracks  12 
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Troops  and  Equipment: 
System  status:  Green  Bar 


0 


Launcher/Missile  Status 


0 


The  average  reported  time  to  complete  crawl  phase  of  training  was  34  minutes. 


Table  D-2 


Walk  Phase  Training  Module  In-Route  Assessments 


Module 

Multiple 

Aircraft 

Attacking 


Decision/Factors  nof  19 

Rank  1  Rank  2  Rank  3 


Decision 

Track  129  13 

Tracks  120,  121,  and  122  7 

Track  1 17  2 

Track  113  5 


Factors 

Mission: 


No  changes;  attack  underway  3 

ADW  RED  9 

Alert  State  1  4 

WEAPONS  HOLD  1 

Enemy: 

Track  data  for  Track  129;  IFF:  No  Response  ||f 

Track  data  for  Tracks  120,  121,  and  122;  IFF:  No  § 

Response 

Track  data  for  Track  1 17;  IFF:  No  Response  2 

Track  data  for  Track  113;  IFF:  No  Response  §jj 

Troops  and  Equipment: 

System  Status:  Green  Bar  0 

Launcher/Missile  Status:  All  Green  0 


3 

8 

i 

4 


1 

2 

4 

1 
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Probable 

Enemy  Actions 

Soldier  asked  to  rate  ‘probability’: 
l=Highly  Probable;  2=Possible;  3=Improbable 

Track 

Split 

Launch 

ARM 

n  of  19 

Strafe/Bomb 
Targets  in 
AA  Dog 

Hostile  Tracks: 

121,  122,  131,  132,  &  135 

Highly 

Probable 

=  6 

Highly 
Probable 
=  2 

Highly 

Probable 

=  9 

Possible 
=  4 

Possible 
=  7 

Possible 
=  5 

Improbable 
=  6 

Improbable 
=  7 

Improbable 
=  3 

129  (3) 

Highlf 

Probable 

=  8 

Highly 
Probable 
=  9 

Highly 
Probable 
=  7 

Possible 
=  5 

Possible 
=  7 

Possible 

=  6 

Improbable 
=  4 

Improbable 
=  1 

Improbable 
=  3 

113 

Highly 
Probable 
=  2 

Highly 
Probable 
=  2 

Highly 
Probable 
=  5 

Possible 
=  3 

Possible 
=  7 

Possible 
=  6 

Improbable 
=  11 

Improbable 
=  7 

Improbable 
=  7 

Factors: 

Mission: 

Factors  and  Alerts 

6 

_ 

_ 

Enemy: 

Track  data  for  Tracks  121,  122,  131,  132,  and  135 

16 

- 

- 

Track  data  for  Tracks  129 

if 

— 

- 

Track  data  for  Track  117 

2 

- 

- 

Track  data  for  Tracks  113 

■ 

— 

— 

Troops  and  Equipment: 

System  Status:  Green  Bar 

1 

- 

- 

Launcher/Missile  Status:  All  Green 

1 

_ 

_ 
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Decisions  and 

n  of  19 

Factors: 

SCUDs  and  an 
Air  Launch 

Rank  1 

Rank  2 

Rank  3 

Priorities: 

Tracks  137,  138,  139,  and  140 

4 

11 

2 

Track  142 

18 

0 

0 

Track  129 

5 

2 

1 

Track  113 

2 

0 

2 

Classification  of  TBMs: 

SS-21  Scarab  B 

2 

- 

- 

Hwasong  5  (SCUD  B  variant) 

7 

- 

- 

Hwasong  6  (SCUD  C  variant) 

8 

— 

— 

No  Dong  1/No  Dong  A 

4 

— 

— 

ARM  or  ASM  indicated  bv  Track  142: 

C-802KD,  KD-88  (CSS-N-8  Saccade)  -  ASM 

2 

- 

- 

Kh-23  (AS -7  Kerry)  -  ASM 

1 

- 

- 

Kh-58EM  (AS-1 1  Kilter)  -  ARM 

6 

- 

- 

Kh-31P  (AS-17  Krypton)  -  ARM 

■ 

Factors: 

Mission: 

ADW  Red 

4 

_ 

__ 

Alert  State:  1 

0 

- 

- 

WEAPONS  HOLD 

1 

- 

- 

Enemy: 

Track  data  for  Tracks  137,  138,  139,  and  140;  IFF: 

Id 

No  Response 

Track  data  for  Track  129;  IFF:  No  Response 

m 

- 

- 

Track  data  for  Track  142;  IFF:  No  Response 

15 

— 

— 

Track  data  for  Track  113;  IFF:  No  Response 

i 

- 

- 

Troops  and  Equipment: 

System  Status:  Green  Bar 

0 
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Launcher/Missile  Status:  All  Green 


0 


The  average  reported  time  to  complete  the  walk  (intermediate)  phase  of  training  was  36 

minutes. 

Table  D-3 

Run  (Advanced)  Phase  Training  Module  In-Route  Assessments 

Module 

Decision/F  actors 

Rank  1 

n  of  17 

Rank  2 

Rank  3 

Training 

Scenario  1 

Problem  1: 
Multiple 
Aircraft 
Attacking 

Priority  of  Threats  to  the  Mission: 

Inbound  ABTs 

9 

4 

2 

Jammers  and  ECM 

8 

5 

3 

TBMs 

7 

3 

7 

Highest  Priority  Action: 

Prepare  to  engage  ABTs 

3 

Prepare  to  engage  the  inbound  jammers  only 

0 

- 

- 

Implement  all  available  Patriot  Electronic  Counter¬ 
countermeasures  (ECCM) 

15 

Factors: 

Mission: 

No  change;  air  attacks  are  underway  and  missile 
attach  warnings  have  been  issued,  targets  are 
US/Coalition  air  and  air  defense  forces 

1 

- 

- 

ADW  Red 

2 

— 

— 

Alert  State:  1 

0 

— 

— 

WEAPONS  HOLD 

0 

— 

— 

Enemy: 

Data  for  Tracks  041  and  043;  IFF:  No  Response 

if 

Data  for  Tracks  045  and  046;  IFF:  No  Response 

15 

- 

- 

Data  for  Tracks  042  and  044;  IFF:  No  Response 

14 

- 

- 

Troops  and  Equipment: 

System  Status:  Green  Bar 

1 

Launcher/Missile  Status:  All  Green 

0 

Problem  2: 

Rank  1 

Rank  2 

Rank  3 

Multiple 

Aircraft 

Attacking 

Prioritized  Threats: 

Tracks  047,  048,  049,  050,  051,  052,  053,  and  054 

3 

11 

2 
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Tracks  041  and  043  2 

Track  055  17 


Factors: 

Mission: 

No  change;  air  and  missile  attacks  are  underway  2 

ADW  Red  0 

Alert  State:  1  0 

WEAPONS  HOLD  0 

Enemy: 

Data  for  Tracks  047,  052,  053,  and  054;  IFF:  No  13 

Response 

Data  for  Tracks  048,  049,  050,  and  05 1 ;  IFF:  No  U 

Response 

Data  for  Tracks  041  and  043;  IFF:  No  Response  6 

Data  for  Track  055;  IFF:  No  Response  |jj 

Troops  and  Equipment: 

System  Status:  Green  Bar  0 

Launcher/Missile  Status:  All  Green  0 


2  11 

0  0 


Problem  3: 

Priority 

Priority 

Maintenance 

1 

2 

and  Reload 

Maintenance  and  Reload  Priorities: 

Prepare  to  conduct  reload  operations  on  LSI  and  LS3 

2 

9 

Reload  LS 1 

3 

0 

Reload  LS3 

2 

3 

Address  the  hazard  conditions  on  LS  2 

■ 

2 

Factors: 

Mission: 

No  changes;  AA  DOG  remains  the  primary  protected 
a  radar  is  to  reorient  to  35  degrees;  threat  anticipated 
from  BM  7,  and  9. 

11 

- 

ADW  RED 

2 

— 

Alert  State:  1 

1 

— 

WEAPONS  HOLD 

1 

— 

Enemy: 

Track  data  for  Tracks  008  and  009;  IFF:  No  Response 

13 

Track  data  for  Tracks  010,  011,  012,  and  013;  IFF: 

No  Response 

12 

— 

Troops  and  Equipment: 

System  Status:  Amber 

$ 

Launcher/Missile  Status:  2  Amber 

6 

- 

Training 
Scenario  2 


Problem  1: 

Rank  1 

Rank  2 

Rank  3 

Assessment  of 
Current  Threats 
and  Probable 
Actions 

Probability  Ranking  of  Threat: 

ARMs  launched  by  inbound  MiG-29s  and  MiG-23s 

12 

3 

2 

Jamming  from  MiG-29s 

2 

1 

5 
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i 

5 


Harpy  UCAVs  5  5 

TBMs  from  BMOAs  3, 7,  &  9  2  6 

Hostile  Tracks  or  Capability: 

008  &  009:  11 

Attack  Your  Battery  &  Patriot  Radar 

008  &  009:  | 

Attack  Maneuver  Forces  in  AA  DOG 

010,  011,012,  &  013:  | 

Attack  Your  Battery  &  Patriot  Radar 

010,  011,012,  &  013:  9 

Attack  Maneuver  Forces  in  AA  DOG 

014,  015,016,  017,  018,  &  020:  H 

Attack  Your  Battery  &  Patriot  Radar 

014,  015,016,  017,  018,  &  020:  8 

Attack  Maneuver  Forces  in  AA  DOG 

022  &  023:  11 

Attack  Your  Battery  &  Patriot  Radar 

022  &  023:  I 


Attack  Maneuver  Forces  in  AA  DOG 


Factors: 

Mission: 

No  changes;  reorientation  of  radar;  missile  and  air 
attacks  against  U.S. /Coalition  air  and  air  defense 
expected 
Enemy: 


Track  data  for  Tracks  008  &  009  §j| 

Track  data  for  Tracks  010,  011,012,  and  013  8 

Track  data  for  Tracks  014,  015,  016,  017,  018,  &  020  12 

Track  data  for  Tracks  022  &  023  § 

Troops  and  Equipment: 

System  Status:  Green  0 

LS  3  out  of  action  for  reload  4 

Time: 

10  minutes  until  LS  3  is  reloaded  and  ready  to  fire  4 


Problem  2: 

Multiple  TBMs 
on  Scope 

Identify  Highest  Probability  Action: 

Designate  the  initial  volleys  of  the  seven  TBMs  (032,  13 

033,  034,  035,  036,  037,  &  038)  for  engagement 
Specify  the  follow-on  volleys,  three  TBMs  (046,  047,  3 

&  048)  for  engagement  hold 

Await  additional  IFF  confirmation  of  the  ten  TBM  3 

tracks  as  hostile  before  designating  them  for 

engagement 

Cease  engagements  until  the  IFF  interrogation  0 

capability  is  restored  and  the  HAZARD  fault  is 
cleared 

Factors: 

Mission: 

No  change;  missile  attacks  are  underway 
ADW  RED 
Alert  State:  1 
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2 

1 

1 


WEAPONS  HOLD 

Enemy: 

1 

— 

— 

Data  for  Tracks  032,  033,  036,  037,  &  038;  IFF:  No 
Data 

13 

— 

— 

Data  for  Tracks  034  &  035;  IFF:  No  Data 

8 

- 

- 

Data  for  Track  046;  IFF:  No  Data 

6 

— 

— 

Data  for  Tracks  047  &  048;  IFF:  No  Data 

2 

— 

— 

Troops  and  Equipment: 

System  Status:  Amber  =  both  TARGET 

EVALUATION  and  TGTID  (IFF)  are  DEGRADED; 

9 

- 

- 

a  HAZARD  is  detected  on  LS  2 

Launcher/Missile  Status:  2  RED 

4 

Problem  3: 
Quickly 

Recover  Mission 
Capabilities 

Identify  Reload  Priorities: 

LS  1 

■ 

2 

LS  2 

6 

0 

— 

LS  3 

4 

5 

— 

LS  4 

0 

1 

-- 

Factors: 

Mission: 

Recover  for  continued  operations;  reorient  radar  to 

360  degrees;  prepare  to  continue  the  mission;  defend 
against  air  and  missile  attacks 

11 

- 

- 

ADW  RED 

3 

— 

— 

Alert  State:  1 

1 

— 

— 

WEAPONS  HOLD 

Enemy: 

0 

— 

— 

No  hostile  tracks  on  scope 

Troops  and  Equipment: 

1 

— 

System  Status:  Amber  -  both  TARGET 

EVALUATION  and  TGTID  (IFF)  are  DEGRADED; 
a  HAZARD  is  detected  on  LS  2 

■ 

Launcher/Missile  Status:  2  RED 

| 

— 

— 

Training 

Scenario  3 

Problem  1: 

Priority 

Priority 

Priority 

Multiple 

Aircraft 

Attacking 

Prioritized  Threats  to  Your  Mission: 

1 

2 

3 

Inbound  massed  formation  of  ABTs  (Tracks  007,  008, 

4 

2 

3 

009,  010,  &  011) 

Maneuvering  flight  (Tracks  015[ACR]  &  014) 
heading  S 

i 

5 

4 

Maneuvering  flight  (Tracks  016  &  017)  heading  SSW 

7 

2 

1 

Inbound  UNKNOWNS  (Tracks  018  &  019) 

10 

1 

2 

Probable  Tvne  of  Missiles  for  Tracks  018  &  019: 

AS -7  Kerry  ASM 

1 

KD-88  Saccade  ALCM 

4 

— 

— 

AS-11  Kilter  ARM 

3 

— 

— 
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AS-17  Krypton  ARM  9 

Factors: 

Mission: 

No  change;  air  attacks  underway;  ASMs  launched  1 

ADW  RED  1 

Alert  State:  1  1 

WEAPONS  HOLD  0 

Enemy: 

Data  for  Tracks  007,  008,  009,  010,  &  01 1 ;  IFF:  No  || 

Response 

Data  for  Tracks  014  &  015;  IFF:  No  Response  13 

Data  for  Tracks  016  &  017;  IFF:  No  Response  10 

Data  for  Tracks  018  &  019;  IFF:  No  Response  10 

Troops  &  Equipment: 

System  Status:  Green;  LS  3  -  PAC  Canister  Problem  2 

Launcher/Missile  Status:  3  Yellow  0 


Problem  2: 

Priority 

Priority 

Priority 

Multiple 

Aircraft 

1 

2 

3 

Attacking 

Crew  Task  Prioritization: 

Prepare  to  engage  the  inbound  TBMs  (Tracks  026, 

| 

4 

4 

027,  028,030,  031,  &  032) 

Prepare  to  engage  the  ALCMs  (Tracks  034  &  035) 

9 

1 

2 

Prepare  to  engage  the  “leakers”  from  the  massed 
formation  (Tracks  007,  009,  010,  &  01 1) 

7 

4 

6 

Factors: 

Mission: 

Air  and  missile  attacks  are  underway 

2 

ADW  RED 

0 

Alert  State:  1 

1 

WEAPONS  HOLD 

2 

Enemy: 

The  locations  and  actions  of: 

ABT  Tracks  007,  009,  010,  &  01 1 

15 

ABT  Track  013 

4 

TBM  Tracks  026,  027,  028,  030,  031,  &  032 

15 

ALCM  Tracks  034  &  035 

13 

Troops  &  Equipment: 

System  Status:  Green  Bar 

0 

Launcher/Missile  Status:  3  Yellow 

0 

Problem  3: 

Multiple 
Inbound  TBMs 

Decision  Priorities: 

Prepare  to  engage  Tracks  035  &  036 
Prepare  to  engage  Track  013 
Begin  reloading  procedures 

Factors: 


10  5  2 
5  §  5 
7  3  6 


Mission: 

Missile  attack  is  underway 
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1 


ADW  RED  0 

Alert  State:  1  2 

WEAPONS  HOLD  2 

Enemy: 

Track  data  for  Tracks  027,  031,  &  032;  IFF:  No  8 

Response 

Track  data  for  Tracks  026  &  030;  IFF:  No  Response  3 

Track  data  for  013;  IFF:  No  Response  9 

Track  data  for  Tracks  035  &  036:  IFF:  No  Response  1 1 

Troops  &  Equipment: 

System  Status:  System  GREEN;  LS  3  has  a  PAC  3  6 

Misfire 

Launcher/Missile  Status  3 


The  average  reported  time  to  complete  the  run  (advanced)  phase  Module  1  of  training 
was  23  minutes;  Module  2,  28  minutes;  and  Module  3,  12  minutes.  The  total  average  reported 
time  to  complete  the  three  Run  phase  modules  was  63  minutes. 


D-12 


APPENDIX  E 

Post-Training  Survey  for  the  Patriot  Cognitive  Skills  Trainer 
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Post-Training  Survey  for  the  Patriot  Cognitive  Skills  Trainer 


Part  I.  Answer  the  following  questions  based  on  your  experiences  with  the  Patriot  Cognitive 
Skills  Trainer. 

1.  The  Cognitive  Skills  Trainer  provided  models  or  methods  for  problem  solving  and 
engagement  problems: 

YES: _  NO: _ 

2.  The  Cognitive  Skills  Trainer  provided  scenarios  and  situations  to  assess  my  critical  thinking 
and  problem  solving. 

YES: _  NO: _ 

3.  The  Cognitive  Skills  Trainer  provided  a  means  to  practice  my  critical  thinking  and  problem 
solving  skills. 

YES: _  NO: _ 

4.  The  scenarios  presented  were  realistic. 

YES: _  NO: _ 

5.  The  scenarios  presented  were  challenging. 

YES: _  NO: _ 

6.  The  scenarios  presented  were  frustrating. 

YES: _  NO: _ 

7.  The  scenarios  presented  were  too  difficult. 

YES: _  NO: _ 

8.  The  scenarios  presented  were  too  easy. 

YES: _  NO: _ 
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Instructions:  The  matrices  below  permit  you  to  self-assess  your  pre-  and  post-training 
understanding  and  skills.  In  the  matrix  below,  rate  your  before-training  skills  and  your  after¬ 
training  skills  as  these  skills  relate  to  critical  thinking  and  decision  making  in  uncertain 
situations.  In  the  spaces  provided  above  the  scale  place  a  “B”  to  indicate  your  level  of  skill  and 
understanding  BEFORE  training  with  the  Tool.  Additionally  in  the  space  provided  below  the 
scale,  place  an  “A”  to  indicate  your  level  of  skill  and  understanding  AFTER  experiencing 
training  with  the  Tool.  See  the  example  below: 


EXAMPLE 

Before  "B" 

B 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

A 

Considering  your  previous  training,  to  what  degree  did  the  desktop  Patriot  Cognitive  Skills 
Trainer  improve  on  your  understanding  of  the  following? 

9.  Setting  work  and  task  priorities 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

10.  Reviewing  and  modifying  priorities 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 
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11.  Setting  engagement  priorities  for  tracks  or  targets  on  my  scope 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

12.  Self-assess  your  understanding  and  skills  when  reviewing  Patriot  system  data  and 
anticipating: 

12a.  Track  separations 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

12b.  Launch  of  ASMs  or  ARAMs 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

12c.  Jamming 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 
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12d.  The  intent  of  a  track  or  a  formation  of  hostile  aircraft 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

12e.  The  need  to  alter  system  settings  on  the  Patriot 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

12f.  The  probable  sequence  of  events  during  a  hostile  attack 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 
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13.  Self-assess  your  understanding  and  skills  when  reviewing  Patriot  system  data  and 
acting  to  counter: 

13a.  Erroneous  classification  of  tracks 


Before  "B" 

■ 

■ 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

13b.  Coordinated  SEAD  directed  against  my  Patriot  system  and  my  battery 


Before  "B" 

Scale 

0 

1  2 

3 

4 

5 

6 

7 

Associated 

phrase 

No 

experience 

Can  get  by  Pretty  good 

Can  train 

others 

After  "A" 

14.  Please  briefly  describe  three  ways  you  might  apply  to  your  job  what  you  learned  from  using 
the  Cognitive  Skills  Trainer? 

(a) 

(b) 

(c) 

14b.  Of  the  three  ways  you  anticipated  using  what  you  learned  (listed  above),  which  would  be 
the  most  important  for  us  to  focus  on  when  further  developing  the  Cognitive  Skills  Trainer? 
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Training  Module  Rating  Questionnaire 
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Training  Module  Rating  Questionnaire 

INSTRUCTIONS:  Please  indicate  the  degree  to  which  you  are  in  agreement  with  each  of 
the  statements  below. 


Item 


Strongly 

Disagree 


(Quality  of  Learning  Experience) 

1 .  I  would  recommend  that  this  training  module  be  0 

made  available  to  all  Patriot  ECS  crews. 

2.  I  would  use  this  training  module  to  refresh  my  skills  1 

at  a  later  date. 

3.  I  feel  I  have  a  better  understanding  of  the  task  after  0 

completing  the  training  module. 

4.  I  preferred  this  training  module  to  others  I  have  used  1 
in  the  past. 

5.  The  training  module  interactively  helped  my  1 

learning  process. 

6.  On  the  basis  of  this  training  module,  I  could  execute  0 

the  task(s)  trained  as  part  of  an  ECS  crew. 

7.  I  feel  this  training  module  was  able  to  meet  my  1 

individual  learning  needs. 

(Quality  of  Design  and  Content) 

8.  The  displays  on  the  screen  were  clear  and  legible.  0 

9.  The  graphics  supported  the  material  being  0 

presented. 

10.  Prompts  and  cues  in  the  training  module  assisted  me  0 
in  navigating  through  the  material. 

1 1 .  The  information  presented  seemed  accurate  and  0 

doctrinally  correct. 

12.  I  felt  like  I  was  in  control  of  my  learning  process.  0 

13.  The  information  presented  seemed  up-to-date.  1 

14.  I  could  easily  track  where  I  was  in  the  training  0 

module. _ 

(Continuity  of  Topics) 

15.  There  was  a  good  connection  between  the  topics.  0 

16.  The  sequence  of  topics  seemed  to  build  on  each  0 

other. 

17.  Training  module  content  was  grouped  to  facilitate  1 

learning. 

18.  There  was  a  clear  focus  of  topics  in  the  training  0 

module. 

19.  Grouping  of  content  allowed  me  flexibility  in  0 

accessing  material. 

(Credible  Examples) 

20.  Examples  contributed  to  my  learning.  1 

21.  The  examples  made  sense.  1 

22.  I  learned  a  lot  about  the  task  from  the  examples.  1 

23.  Examples  were  presented  in  a  realistic  mission  1 

context. 


Number  of  Participants 
(Out  of  20) 

Disagree  Neither  Agree 
Agree 
or 

_ Disagree _ 


3  2  8 

2  1  9 

2  5  8 

2  6  7 

3  1  13 

1  8  9 

4  3  9 


0  1  12 

0  1  12 

2  2  9 

2  5  9 

2  4  6 

3  2  10 

0  2  14 


0  1  12 

0  1  13 

2  2  12 

1  2  10 

0  4  9 


2  2  9 

1  3  10 

1  3  11 

2  3  9 


Strongly 

Agree 


7 

7 

5 

3 

2 

2 

2 


7 

7 

7 
4 

8 
4 
4 


6 

6 

3 

6 

7 


6 

5 

3 

5 
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24.  Repetition  of  examples  was  helpful. 

0 

0 

6 

8 

6 

(Focus  and  Relevance) 

25.  Sections  of  the  training  modules  were  of  the  right 
length  to  allow  me  to  complete  them  without 
needing  a  break. 

0 

0 

2 

12 

6 

26.  Questions  asked  within  the  training  modules  were 
reasonable  and  helped  me  to  understand  the  topic. 

0 

1 

4 

10 

5 

27.  The  questions  asked  within  the  training  module 
focused  on  what  was  being  taught. 

0 

0 

4 

10 

6 

28.  The  overall  focus  of  the  training  module  was  right 
on  target. 

(Tracking  Progress) 

1 

1 

2 

8 

6 

29.  If  I  took  a  break  during  the  learning  process,  I  could 
easily  resume  learning  when  I  returned. 

0 

0 

0 

11 

9 
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